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Introduction 


FRANZISKA HARTUNG' & BUDDHIKA BELLANA® 


He spend a considerable amount of time engaging with narratives and art. Some thinkers 
propose that our minds/brains are optimized for information in the form of narratives (Mar et 
al.), going so far as to invoke terms such as homo narrans (Fisher) or homo aestheticus (Dissanayake). 
Others claim that our experience of our own lives is inherently narrative in character (Bruner; 
Schank & Abelson). Works of art can shape cultures and elicit powerful emotional responses — 
responses that may be difficult to elicit otherwise (Robinson). We have a profound sense that narra- 
tives and art are an important aspect of being human. Why do narratives and art have such a hold 
over us? What might this affinity tell us about the architecture of our minds and brains? 

Over the past 15 years, empirical research on literature, poetry, drama, arts, film, and dance have 
begun to gain a foothold in cognitive neuroscience. As cognitive neuroscientists, we have come to 
learn that our models of language, memory, emotion, and perception fall short of providing satisfac- 
tory accounts of our aesthetic experiences with narratives and art (Chatterjee & Vartanian; Lee, 
Bellana & Chen; Goldstein & Bloom; Nastase et al.,; Hartung et al.,; Hasson et al.; Sonkusare et al.; 
Willems). Feeling immersed in a story or song, appreciating the beauty ofa painting, or reveling in 
the lasting impact ofa film or play are not trivial aspects of human experience — instead they lie at the 
heart of some of our deepest questions. As such, this gap between cognitive neuroscience and 
aesthetics has become a fertile ground for empirical development and discovery. 

In this special issue, we want to pay tribute to recent advances in understanding the human mind 
and brain through narratives and art, while also highlighting current challenges and charting out 
plans for the future. The contributions from cognitive and neuro- sciences, humanities and media 
studies, together reflect a refreshing interdisciplinary dialogue and, hopefully, a compelling window 
into recent work on understanding humans through narratives and art. 


On narratives 


In our first section on narratives, we present three contributions from cognitive neuroscience and 
linguistics. 

We begin with a discussion of how narratives can inform our understanding of the neural bases of 
human memory. Cohn-Sheehy proposes the idea of an ‘intertextual hippocampus’, carrying for- 
ward the basic associative functions often ascribed to the hippocampus, a brain structure tucked into 
our medial temporal lobes, to how we remember naturalistic experiences, such as the events in a story. 

Next, we explore individual differences across readers in their ability to take the words from a 
story and simulate a story world. Drawing from behavioural and brain research, Mak & Faber 
propose the importance of mental simulation in predicting the extent to which an individual will 
find a story immersive and enjoyable. 

Lastly, narrative comprehension requires readers to track situational information like the conti- 
nuity of characters, locations, or events, while additionally requiring us to make inferences from the 
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broader context. Klomberg and colleagues present a novel framework addressing how this may be 
accomplished in visual narratives, such as comics, in which these inferences are supported by visual 
constructional patterns. This piece highlights the fundamental role of graphic and structural cues in 
visual storytelling. 


On art and empathy 


The second section of this special issue tackles recent work at the intersection of art and empathy. 

Narratives have long been connected with empathy. In her position piece, Turner reviews the 
latest research on narratives and empathy and argues that both empathy and fiction are ontologically 
extended processes and highlights the importance of taking cultural aspects and different fiction 
technologies into account in future research. 

Next, Woodward & Woodward take a truly interdisciplinary approach to the interconnectedness 
of pain, stress, and suffering. They discuss perspectives on pain, stress and suffering from both a 
literature and a neuroscience perspective. They also address how to quantify and qualify these 
experiences alongside the self-understanding that is needed to assess them. 


On aesthetics 


In our third and final section, we present recent interdisciplinary work on aesthetics, or how and 
why we appreciate art. 

First, we turn to Torromino and colleagues who review work on the hypothesis that art should be 
treated as supernormal stimuli. They propose an integrated perspective bridging art with neuro- 
science to show the potential that this conceptualization, rooted in ethology, can bring to our 
understanding of aesthetics. 

Next, we examine the aesthetics of reading. Trasmundi & Kukonnen make an argument for 
reviving our interest in ‘aisthesis’ - embodied sensation and perception - in the study of reading. In 
a proof of concept study, they describe the impact of aesthetic practices in situated embodiment for 
processing and appreciating a classic piece of literature: Goethe’s Faust. Using this approach, they 
showcase the potential of this embodied cognition framework in the study of aesthetics in reading. 

Importantly, any examination of aesthetics is incomplete without a consideration of the cultural 
context in which we are embedded and how it may shape how we appreciate art. To this end, Darda 
& Chatterjee present an empirical study examining aesthetic preferences in a cross-cultural com- 
parison of people from North America and India. They discuss the potential of education about and 
exposure to art from different cultures in helping to reduce social bias. 

Finally, we explore the impact of artificial intelligence on art and aesthetics. In their piece, Abiodun 
& Nickel provide a timely dialogue between artist and scientist, as they meditate on how art and 
artificial intelligence continuously influence one another, in often unexpected ways. 


Concluding thoughts 


In this special issue, we take a glimpse into the study of the human mind and brain via narratives 
and art. We address a wide range of topics, from how stories can inform our neural models ofhuman 
memory to examining how artificial intelligence may change our understanding of art itself. We 
hope that our readers find this special issue thought-provoking and useful. 

One idea that has become plain over the course of preparing this special issue, is that neuroaesthetics 
—and the broader study of narratives and art — remains a well-spring of exciting questions about the 
human mind and brain. A key reason why this line of inquiry continues to be so generative is its 
inherent interdisciplinarity. Neuroaesthetics implies that a basic understanding of the brain, and our 
subjective experience, is incomplete without thoughtful consideration from the humanities. Simi- 
larly, our understanding of art and aesthetics is only deepened with rigorous experimentation about 
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its basic properties and neural implementation. Not only is there space for both scientists and hu- 
manities scholars in the field, but neuroaesthetics plainly highlights the shortcomings of either ap- 
proach when pursued in isolation. What is a model of vision that has nothing to say about the 
experience of beauty drawn out by painting? What is a framework of language that has nothing to 
say about our ability to evoke alternate worlds from the words in a novel? What is a theory of 
aesthetic experience with no consideration of the machinery upon which our minds are built? 

Stories, poetry, drama, arts, film, and dance are all fundamental parts of the human experience. If 
we are interested in anything that resembles a complete understanding of how the human mind and 
brain works, neuroaesthetics will be a useful, if not essential, means towards this end. 


‘Newcastle University, UK 
* York University, Glendon Campus, Toronto, Canada 
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Narrative Memories Woven by an Intertextual 
Hippocampus 


BRENDAN I. COHN-SHEEHY 


Abstract: Narratives fundamentally shape the way we remember real-life experiences. However, 
neuroscientists have only begun to understand how narratives impact the way our brains support 
memory. In this opinion piece, I illustrate how the hippocampus, a key region of the brain for 
memory, transforms our experiences into larger narratives in memory. Furthermore, I argue that 
the hippocampus provides a biological basis for “intertextuality” — that is, all experiences or texts may 
be necessarily understood and remembered in relation to other experiences or texts. An intertextual 
hippocampus has tangible consequences for our lives and our art. 


Keywords: narratives, memory, intertextuality, neuroscience, hippocampus 


Introduction 


O memories often take the form ofa larger story which can be elicited with a reminder. One 
personal reminder is a tapestry on my wall, an abstract geometric array woven from black and 
rust orange yarn. This is my grandmother’s tapestry, “Walk of Fame,” and its appearance readily 
conjures a series of interrelated scenes scattered throughout time. I remember curiously looking 
over the half-finished tapestry on my grandmother’s room-sized loom. Seeing my interest, she later 
gave me my first lesson in weaving, showing me how to use a device called a shuttle to pull threads 
of yarn across the loom, and then to pull certain levers to intertwine those threads into a textile. I can 
then envision the room where the loom once stood, vacant except for a closet full of yarn, following 
her death; my spouse’s aunt handing me a beautiful rust orange scarf, knitted from my late 
grandmother’s yarn; and that scarf, sealing my warmth on a cold morning walk to work, a daily 
reminder of my antecedents’ “walk of fame.” 

As a neuroscientist, I have endeavored to understand how the brain can form such intricate 
impressions of the past in our memory. I think that weaving is an apt metaphor for how memory 
works. If you consider the brain to be a kind of loom, there are ways in which our ongoing experi- 
ence gets “shuttled” through this loom like threads of yarn. There are also “levers” which get pulled, 
such that these threads become intertwined into a recognizable form. The product, memory, is a 
tapestry in which the timeline of our experience becomes warped, and experiences that were ini- 
tially separated in time are brought together to form a larger story. 

In this opinion piece, I hope to illustrate how the way we translate experience into memory is 
profoundly shaped by narratives. Although this is not an exhaustive review, I provide some context 
about the science of memory, before elaborating on recent findings about how narratives impact 
memory. I describe how the hippocampus, a key brain structure for memory, incorporates past 
memories into our ongoing experience, enabling us to build coherent narratives from events that are 
otherwise separated in time. I then argue that the way this works suggests that the hippocampus 
supports a kind of “intertextuality” — that is, new memories are created with reference to other 
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memories, and no memory truly stands alone from other memories. Finally, I speculate on the 
implications of an intertextual hippocampus. 


The science of isolated events 


Even if our experiences become part of a larger tapestry in memory, the neuroscience of memory 
has historically tried to understand how we remember each individual experience. For instance, try 
to remember what you had for dinner a couple nights ago. In my case, the other night, I had braised 
noodles with leeks and crab meat. I can imagine myself lifting the noodles to my mouth with 
chopsticks from a white plate, and then savoring the contents. In the background, I can also envision 
my spouse and a few take-out trays at our dining table. That is, my brain allows me to not only access 
the contents of my dinner, but also other associated details from that time which allow me to relive 
the experience of that specific meal. 

This capacity to store and retrieve information that is encountered within specific, time-limited 
“events”, is termed episodic memory (Tulving, “Episodic and Semantic Memory”). Seventy years of 
research has established that episodic memory depends on the hippocampus, a seahorse-shaped 
structure embedded deep within the brain (see Figure 1). In the 1950s we learned that if you surgi- 
cally remove someone’s hippocampus, they cannot form memories for new events (Scoville and 
Milner). This kind of deficit was later discovered in many other patients with injury to the hippoc- 
ampus, and indicated that the hippocampus is necessary for episodic memory (Mishkin et al.; Tulving, 
“Episodic Memory”; Yonelinas et al.). 


Parietal 
cortex 


Prefrontal 
cortex Fig. 1: The figure shows “slice” through 
an MRI scan ofan individual's brain, with 
the hippocampus (arrow) traced in dark 

rey. The cortex (brackets) constitutes 
the grey ridges and folds covering the 
surface of the brain’s four lobes (frontal, 


temporal, parietal, occipital). 


Occipital 
cortex 


Temporal 
cortex 


Hippocampus 


Subsequently, our field tried to understand how the hippocampus supports episodic memory. This 
endeavor was supported by the advent of functional magnetic resonance imaging (fMRI), a non- 
invasive technique which allowed us to study how the brain works in healthy individuals without 
any brain damage. The basic way {MRI works is that we have someone lie down inside a giant 
magnet, and by perturbing the magnetic field surrounding that person’s head, we can read off 
energy from within their brain. We can then use this energy to construct a series of 3-dimensional 
“heat maps” which show us how more-or-less active all the parts of someone’s brain are, at various 
moments in time. We can even have someone do different memory tasks during f{MRI, and we can 
then compare the heat maps which correspond to these different tasks to test our hypotheses about 
how the brain supports memory. 

In order to investigate how people can remember specific events, {MRI studies have typically had 
people memorize series of words or images, manipulating the way these words or images are pre- 
sented or tested. Although this approach does not approximate the complex nature of real-life events, 
it has enabled us to conduct well-controlled scientific experiments. 
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fMRI studies have taught us that the hippocampus supports episodic memory by interacting with 
the brain’s cortex, the complex array of ridges and grooves which cover the surface of the brain (see 
Figure). We have learned that during an ongoing experience, various regions of the cortex that are 
connected to the hippocampus support our ability to recognize features like individual people, 
places, objects, and concepts (Ranganath and Ritchey). The way these co-occurring features come 
together to comprise an “event” depends on the hippocampus. Much like on a loom, these features 
are shuttled through the hippocampus and become interwoven to form a memory of the event 
which took place (Diana et al.). 

That said, there is more to weaving than arbitrarily intertwining threads, and memories for 
individual events are not meaningless packages of co-occurring features. For instance, the dinner I 
described did not take place in a meaningless void. Actually, we were eating “longevity noodles” to 
celebrate Lunar New Year. We ordered takeout from our go-to Chinese restaurant because I was 
coming offa long shift at the hospital, and it was too late to find an available table on their busiest 
night. As in this example, some studies have suggested that the way one remembers a real-life event 
can be dependent on one’s understanding of surrounding events (Brown and Schopflocher; Burt et 
al.; Conway and Pleydell-Pearce). 

As we now know, the hippocampus does not merely form isolated memories for each event, but 
also situates them against a backdrop of other events (reviewed in Cohn-Sheehy and Ranganath; 
see also Addis et al.). To grasp how this works, our science has had to move beyond studying 
memory for arbitrary words or images. As I will discuss, narratives have unlocked a greater under- 
standing of memory. 


Narratives shape memory for events 


Many contributors to this journal may find it intuitive that narratives would have a profound 
influence on memory. However, science progresses incrementally, and our empirical understand- 
ing of narratives and memory has had to develop stepwise from basic questions. 

Perhaps the most basic question we could ask was, do narratives impact memory at all? This might 
seem absurd, but try to memorize a series of vague sentences: 


One happens at one time, and one happens later. 
They become a larger whole. 

Putting them together requires one particular region. 
Bringing them back later involves the same region. 


It might be challenging to remember these sentences in their present form. However, if I gave you 
the title, “The hippocampus weaves events into narrative memories,” you might find it easier to 
understand and remember these sentences. Bransford and Johnson had people memorize similarly 
vague series of sentences, and people who were given a title did a much better job of recalling the 
sentences than people who were not given a title (Bransford and Johnson). This suggested that the 
opportunity to form a larger narrative provides a useful basis for remembering smaller pieces of 
information like sentences. 

Narratives not only shape our memory for sentences, but also more realistic events. We know this 
from subsequent work which investigated how people remember longer stories (Trabasso and Sperry; 
Trabasso et al.). Stories offered a more realistic way of studying memory, in part because, much like 
real life, they tend to encompass an extended series of events. 

However, not all events ina story are equally important for forming a narrative. For instance, in 
many stories, there are events which deviate from the main plot as “side events,” and contribute less 
to one’s understanding of the larger story. Trabasso et al. found that when people memorized stories, 
they tended to leave out many, if not all, details from side events, while preferentially recalling main 
plot events (Trabasso et al.; Trabasso and Sperry). That is, people were more likely to forget the 
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events that were less relevant to a larger narrative. This finding suggested that memory for events 
might depend on narrative coherence: the degree to which individual events can be interrelated within 
a single narrative (Graesser et al.). 

At first blush, it seems reasonable that when we follow the plot of a story, we tend to find it harder 
to remember unrelated side events. On the other hand, some events that are unrelated to events 
nearby in time can also be highly related to events that are distant in time. For instance, early in one 
episode of the sitcom “Seinfeld” (O’Keefe et al.), the character Kramer is seen holding a picket sign 
outside of a bagel store. Toward the end of the episode, during a dinner with friends, Kramer 
announces that he has to leave dinner to go bake bagels. Although these events are separated in time, 
they coalesce to form one narrative: that Kramer was on strike, and now the strike has resolved. In 
this example, narrative coherence seems to shape our understanding of events that take place at 
distant times. By extension, narrative coherence might also shape how we remember distant events. 

We recently investigated the impact of narrative coherence on memory for distant events (Cohn- 
Sheehy, Delarazan, Crivelli-Decker, et al.). Toward this end, we created short fictional audiobooks 
which each recounted a series of events involving some main character. Critically, there were 
recurring side-characters, who made brief appearances in two separate stories, but these appear- 
ances bore no relation to the surrounding main stories—that is, they were “side-plot events.” Some of 
these side-plot events were written to form one coherent narrative (i.e. like the Kramer example 
from “Seinfeld”). In contrast, other side-plot events were unrelated, and could not easily form a 
coherent narrative. We reliably found that people could recall more details about the side-plot 
events which formed coherent narratives, than the side-plot events which could not (Cohn-Sheehy, 
Delarazan, Crivelli-Decker, et al.). This was even the case when people were asked to recall these 
events a day after they initially heard the stories. 

This finding suggested that, somehow, narrative coherence can bridge the gap between distant 
events in memory. In other words, new and old events might be interwoven to form a larger 
narrative tapestry. In order to understand this tapestry, we had to understand the loom. 


The hippocampus weaves events into narrative memories 


As I mentioned earlier, a preponderance of evidence had suggested that, like a loom, the hippoc- 
ampus somehow weaves together several pieces of information to create an event in memory. If this 
loom just passively receives and intertwines threads of yarn, it might be able to string together a 
series of contiguous events to form a narrative. However, this would not explain the fact that even 
distant events can form one narrative. 

Instead, we suspected that some active lever on this loom might determine whether distant events 
become interwoven in memory. We tested this idea in an {MRI study which employed the same 
fictional stories described above (Cohn-Sheehy, Delarazan, Reagh, et al.). We used {MRI to scan the 
hippocampus while people first heard the stories, and then a day later, while they recalled events 
from the stories using a microphone in the MRI scanner. 

More specifically, for each event in the stories, we used {MRI to characterize an “activity pattern” 
in the hippocampus. You might think of each event’s activity pattern as a square of woven textile 
containing colored shapes. If two different squares of textile, or events, have an orange circle in the 
upper-right corner, you could say that these patterns are similar. These patterns would be much less 
similar if one of the textiles had an orange circle in the upper-right corner, and the other had a blue 
triangle in the lower-left corner. In this fashion, “pattern similarity” provided a proxy for the way 
that the hippocampus was weaving each event in memory. That is, higher pattern similarity might 
indicate that the hippocampus was weaving two events together in memory. 

Because of the way we designed the stories in our study, we could measure pattern similarity 
between two distant events that formed a coherent narrative, and we could also measure pattern 
similarity between two distant events that were unrelated. What we found was that pattern similar- 
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ity was higher when distant events could form a coherent narrative. This finding provided initial 
support for the idea that narrative coherence might determine how the hippocampus weaves memo- 
ries for events. 

However, this was not the full story. We thought that there might be something special about the 
second of two events that form a larger narrative. Rationally, one cannot easily form a narrative 
across two distant events, until encountering the second event. In our stories, the point where people 
re-encountered a recurring character provided the opportunity to start drawing connections be- 
tween distant events. Accordingly, the activity pattern from the second event might hold the key to 
the whole narrative. 

This idea was supported by our data. When someone in our study was later asked to recall both 
events that formed a coherent narrative, their hippocampus “reactivated” the activity pattern from 
the second event. In fact, the degree to which someone reactivated this activity pattern not only 
predicted how well they could remember the second event, but also the first event. This meant that 
the memory which the hippocampus formed for the second event, somehow incorporated informa- 
tion about the first event, allowing both events to be recalled together. In other words, the hippoc- 
ampus interwove these two distant events to form one “narrative memory” (Cohn-Sheehy, Delarazan, 
Reagh, et al.). 

Returning to our loom analogy: when two events can form a narrative, it is not just that similar 
threads are passively shuttled through the loom of memory. Rather, it is at the time of the second 
event that a lever gets pulled, and the newly shuttled threads ofan ongoing memory become inter- 
woven with the threads of a past memory. What results is that the two events which form a narrative 
are, to some degree, inextricably linked. 

Our work demonstrated an important role for the hippocampus in forming narrative memories 
(see also, Milivojevic, Vicente-Grabovetsky, et al.; Milivojevic, Varadinov, et al.; Collin et al.; Race 
et al.). However, as | mentioned earlier, the hippocampus does not act alone in creating memory for 
an event, but also relies on inputs from the brain’s cortex. In fact, many studies have suggested that the 
cortex plays a pivotal role in our ability to understand events in a narrative (e.g. Chen, Leong, et al.; 
Lee and Chen). That said, the way the cortex supports memory for narratives appears to depend on 
how the cortex interacts with the hippocampus (Chen, Honey, et al.; Barnett et al.; Aly et al.). For 
instance, within a story, there are moments when people can reliably perceive that one event has 
ended, and another has begun (Zacks). At these moments, the hippocampus steps up its activity and 
its interactions with the cortex (Baldassano, Chen, et al.; Ben-Yakov and Henson). This is an active 
area of study, but one emerging explanation is that at these moments, the hippocampus might 
retrieve old events or other information from memory which enable one to understand and remem- 
ber a new event (Chen, Honey, et al.; Lu et al.). That is, even if the cortex is continuously shuttling 
threads of meaningful information about an ongoing story through the loom of memory, it is likely 
the hippocampus which serves as the dynamic, lever-pulling aspect of this loom, determining how 
those threads are interwoven. 

As I will now argue, our new understanding of the way the hippocampus constructs narrative 
memories dovetails with a longstanding idea about literature. 


The hippocampus supports intertextuality 


Many literary theorists have proposed that the way people comprehend and create texts is highly 
dependent on how each text relates to other texts, an idea referred to as intertextuality (for a review, 
see Alfaro). This idea has been applied to many kinds of “texts,” including stories, films, music, and 
other art forms. One version of this idea is that no new text is completely original, because it is 
necessarily built upon prior texts and can only be comprehended in relation to other texts. This 
echoes what I have just described about memory and the hippocampus. The way the hippocampus 
creates memories for each event is built upon memories for other events, and dependent on how 
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events are collectively situated within a larger narrative. Intertextuality may be a property of epi- 
sodic memory, with a biological basis in the hippocampus. 

Perhaps the clearest way to illustrate how intertextuality might be playing out in memory, is to 
consider how intertextuality plays out in literature through the use of allusions. For instance, in James 
Joyce’s Ulysses, recognizing that a belligerent pub customer named “the Citizen” is an allusion to the 
“Cyclops” from Homer’s Odyssey transforms what would otherwise be a liquor-fueled argument 
into a death-defying struggle for survival (Joyce, Ulysses). In real-life, not all of us are graced like 
Joyce with constant access to literary allusions. However, we often experience reminders which, 
equivalently, enable us to draw connections between events (Wahlheim and Jacoby; Ross and 
Bradshaw; Jacoby). Ifyou encounter a person or object that you previously encountered in a past 
event, this can serve as a reminder of that past event and provide an opportunity to draw connections 
between the two events. Moreover, by conjuring information about the past, reminders can effec- 
tively embed information about the past within a new memory for the present. This supports an 
intertextual view of memory, in that new memories are necessarily dependent on, and understood in 
relation to, old memories. 

Furthermore, we know that the hippocampus responds to reminders in ongoing events. When 
fMRI experiments present people with a recurring word or image, the hippocampus supports the 
ability to retrieve information about previous events involving that word or image. Through this 
process, the hippocampus can embed information about the previous event into a memory for the 
new event, allowing the two events to become linked in memory (Horner et al.; Zeithamova et al.). 
However, in contrast, other studies have shown that when events share overlapping words or im- 
ages, the hippocampus will effectively keep those events separate in memory (Chanales et al.). That 
is, not all reminders lead the hippocampus to interweave events (see also Stawarczyk et al.). 

Our work provides some insight into the conditions which encourage the hippocampus to inter- 
weave events (Cohn-Sheehy, Delarazan, Reagh, et al.; Cohn-Sheehy, Delarazan, Crivelli-Decker, 
etal.). In the stories we constructed, recurring fictional characters could serve as a reminder of distant 
events involving those characters. When this kind of reminder could lead to the formation of a 
coherent narrative, the hippocampus embedded information about the prior event within the 
memory of the new event. That is, reminders can lead the hippocampus to draw connections be- 
tween events in memory, but this may specifically depend on narrative coherence. This parallels the 
way literary allusions shape the narrative of a new text. 

Intertextuality is not limited to allusions which shape the meaning of new texts. Intertextuality can 
also involve reshaping one’s understanding of older texts in light of newer texts (Alfaro). Although I 
have just illustrated how new memories are shaped by old memories, the reverse can also take place. 
For instance, when you are presented with a new insight about past events, the hippocampus can 
alter your memory for those past events to accommodate the new insight (Milivojevic, Vicente- 
Grabovetsky, et al.). In fact, when you re-watch a film, but with a changed ending, this can even lead 
the hippocampus to incorporate false details into your memory for the original film (Sinclair et al.). 
In other words, in both good and bad ways, the hippocampus might update old memories to fit an 
evolving narrative about a collective set of experiences. This mirrors the idea that the collective set 
of texts we have experienced shapes our understanding of any individual text. 

It is worth noting that these kinds of reminders and insights tend to involve some degree of 
conscious awareness. However, intertextuality can take place even without conscious awareness, 
and many texts contain similar types of narratives even without any conscious source attribution. 
For instance, both James Joyce’s Ulysses and Homer’s Odyssey depict a kind of “hero’s journey,” a 
storytelling format that may have derived from ancient mythology and oral tradition (Campbell 
and Moyers). Ina similar vein, many real-life events unfold in a predictable way, and our memories 
can become shaped by complex knowledge about how these events tend to unfold (Rumelhart; 
Thorndyke; Mandler and Johnson; Pichert and Anderson; Ghosh and Gilboa). For instance, when 
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you go to a restaurant, you can draw from a restaurant “script” to predict that you will first be sat at 
a table, then order food, then receive and eat food, and finally pay and leave (Schank and Abelson). 
Much like reminders, scripts can provide relevant past information which enables one to understand 
anew event. However, in contrast to reminders, scripts are learned across multiple past events rather 
than referencing a single event. Furthermore, scripts do not capture other unique details of an event, 
like who you were with or what you were eating. 

Recent evidence suggests that areas of the cortex which support our ability to understand narra- 
tives, can also support information about scripts like going to a restaurant (Baldassano, Hasson, et al.; 
Reagh and Ranganath). These findings are in line with the idea that the cortex can provide support 
for complex knowledge and concepts which are derived from prior events, but not consciously tied 
to specific past events in memory (e.g. O’Reilly et al.; Gilboa and Marlatte). However, recent evi- 
dence suggests that in contrast to the cortex, the hippocampus supports the ability to construct 
memories which merge information about scripts with other unique details that take place during 
conscious awareness of an event (Reagh and Ranganath). In other words, even if one unconsciously 
draws upon scripts or other past information, the hippocampus may be responsible for weaving this 
information into one’s conscious memory of a new event or text. 

In summary, the hippocampus may provide support for a spectrum of conscious and unconscious 
forms of intertextuality. At one extreme, the hippocampus may support a conscious synthesis of 
information about past and present experiences to form a narrative in memory. At the other ex- 
treme, the hippocampus may even incorporate unconscious forms of past information into the 
conscious memory of a present experience. In either case, the hippocampus provides a mechanism 
by which memories for any, or all, experiences, are shaped by other experiences. To the degree that 
our many experiences form a narrative tapestry, intertextuality may be a property which deter- 
mines how we create and understand this tapestry. I will now speculate about what this means for life 
and for art. 


Implications of an intertextual hippocampus 


At this point, you may be wondering what neuroscience has actually taught us about narratives or 
intertextuality. Fiction writers have long been able to capture how intertextuality manifests in our 
conscious day-to-day experience (e.g. see Joyce, Ulysses). Hip-hop artists have mastered the use of 
lyrical allusions and audio sampling to build intertextual meaning. And introspectively, after one has 
lived many years, it is hard to imagine that any new event will not resemble some prior event or 
predictable script. Is the science simply confirmatory? 

What science has to offer is an objective test of how these things work. As a result of tedious 
experiments, we have empirical evidence to suggest that intertextuality is more than a concept in 
theory, but a biological reality. Any time you experience a new event or text, the intertextual 
machinery of your hippocampus may be conjuring past events or texts, bringing them to bear on 
how you understand and remember each new experience. 

This might have both positive and negative consequences. In some cases, intertextuality may be 
beneficial for memory. Being able to recognize that new and old events form a narrative can make 
it easier to remember those events in detail (Cohn-Sheehy, Delarazan, Crivelli-Decker, et al.). 
Additionally, your previous experience with texts can make it easier to understand and remember 
similar texts. For instance, people with a strong background knowledge of “Star Trek” find it easier 
to memorize new fiction involving “Star Trek” characters (Long and Prat). 

Conversely, when one cannot easily draw connections between old and new events or texts, this 
can negatively impact memory. In the early twentieth century, F.C. Bartlett presented a Native 
American story called “War of the Ghosts” to English participants who had no prior experience with 
Native American literature. When Bartlett asked the English participants to recall the story, they 
forgot many of the original details. Furthermore, the more times they retold the story, the more it 
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became distorted to sound like a typical English war story of that time (Bartlett). In other words, 
when we are unable to refer to a relevant past experience or text, intertextuality might actually lead 
us to form narratives or memories that may sound like they are true but are actually false. 

Ihave mainly focused on how the hippocampus might support intertextuality for a reader. How- 
ever, the same principles may also apply to the side of the author. In fact, we know that beyond its 
role in memory, the hippocampus also supports the ability to imagine possible events that have not 
yet taken place (Addis, Wong, et al.). That is, the ability of authors to construct new texts about 
possible events may also rely on the same intertextual machinery (Benedek et al.). If so, this may have 
positive consequences for the creative process, especially for an author like James Joyce who in- 
tended to create something new by consciously building upon past texts. 

On the other hand, when this same machinery draws upon unconscious sources, it might contrib- 
ute to unconscious plagiarism by authors. In a famous example, George Harrison produced and 
released the song “My Sweet Lord” without realizing that it contained a combination of motifs and 
chord progressions that had previously characterized The Chiffons’ “He’s So Fine”. Memory re- 
searchers have often interpreted this example as a failure of “source monitoring,” a conscious process 
that enables one to discern where a memory came from (Johnson). Source monitoring is largely 
thought to be supported by the prefrontal cortex (Johnson; see also Shimamura; Moscovitch and 
Winocur), an area of the cortex which is known to dynamically interact with the hippocampus in 
various memory processes. However, while a source monitoring account might explain some lack 
of awareness during Harrison’s creative process, it arguably does not account for the intertextual 
synthesis that the creative product represents. Somehow, Harrison incorporated motifs and chords 
from an unconscious source, a love song about infatuation, into the conscious creation of a unique 
spiritual love song about his relationship with God. This type of synthesis may have depended on the 
intertextual machinery of the hippocampus. Furthermore, listeners may not have even recognized 
“My Sweet Lord” as a form of plagiarism without an intertextual hippocampus that reminded them 
of “He’s So Fine.” 

Whether or not intertextuality benefits memory or art, it is pervasive in our lives. Recent work 
from Bellana et al. demonstrated that when we deeply engage with a story, it can linger in mind for 
along time and even influence our spontaneous thoughts (Bellana et al.). Even the seemingly random 
thoughts we have in our day-to-day lives might actually draw upon some literary influence. By a 
logical extension of this finding, it might be impossible to completely separate out thoughts that we 
think are original, from thoughts which reflect a memory for past stories. Intertextuality might be so 
deeply wired into how the brain processes our experiences, memories, and thoughts, that it usually 
goes unnoticed. 

It is worth asking why intertextuality would be a property of the brain. One possibility is that, 
through our upbringing and education, we are so immersed in stories—through literature, films, 
music, and day-to-day conversation—that by some age, our brains have adapted to the task of 
remembering a massive set of interrelated narratives. The more our literature has evolved, the more 
it may have evoked some innate capacity of our brains for intertextuality and narrative memories. 
In turn, by harnessing the power of stories, neuroscience may further elucidate the facts about our 
“uncreated conscience” (Joyce, A Portrait of the Artist as a Young Man). 
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From Words to Worlds: Is Mental Simulation a 
Driver of Individual Differences in Processing, 
Experiencing, and Liking Stories? 


Abstract: Stories allow people to (vicariously) experience other worlds, but what this subjective 
experience looks like varies from reader to reader. In this paper, we focus on the cognitive and neural 
mechanisms that give rise to these differences in experience and preference. We argue that differ- 
ences in reading skill and in the reading “modes” that a reader has access to might be important 
predictors, together with factors such as the reader’s propensity for mental simulation. We discuss 
evidence to suggest that these differences are reflected in behavioural and neural signatures. 


Keywords: Reading, story world absorption, literary appreciation, mental simulation, lexical charac- 
teristics 


So are thought to be a universal aspect of human culture: by providing a narrative context, 
they offer a cognitively ergonomic way of transmitting and receiving information, cultural 
values, and beliefs, and are an important source of entertainment and education (Bruner; Rubin). 
How people process, experience, and like stories, however, appears to be highly dependent on the 
individual: some people for instance enjoy reading complex literary stories that use defamiliarizing 
language, whereas others prefer easy-to-read, suspenseful, and action-packed stories. A recent paper 
showed that, counterintuitively, people who feel more absorbed in a story are also a bit more 
“detached” from the actual words they were reading as measured by eye tracking (Eekhof et al.), 
whereas detachment is usually regarded as a sign of disengagement (Faber, Krasich, et al.). Around 
the same time, it was observed that people vary somewhat systematically in their preferences for 
specific stories (Mak, Faber, et al., “Different Routes to Liking: How Readers Arrive at Narrative 
Evaluations”), but that it is unclear what drives this variation. 

In the current paper, we aim to bring together these different lines of research, and set out to 
explore how different reader characteristics influence how people read, experience, and like stories. 
We ask whether mental simulation—the automatic activation of sensory and emotional information 
during language processing (Mak and Willems, “Mental Simulation during Literary Reading”)— 
might be an explanatory factor: when the brain is constructing internal representations, the external 
environment might be processed less deeply, potentially leading to the observed detachment. At the 
same time, this process might form the basis for or be a driver of people’s experience during reading: 
stronger or richer activations of sensory and emotional information might play a role in how absorbed 
people feel, and how much they like a story. However, these processes might not happen in isolation, 
but rather are likely to be constrained by characteristics of the reader, the text, and other factors. 

The aim of this paper is to provide a (by no means comprehensive) discussion of evidence for the 
idea that there are individual differences in how people read, experience, and like stories, and that 
differences in reading skill and in the reading “modes” that a reader has access to might be important 
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predictors of this variation, together with factors such as the reader’s propensity for mental simula- 
tion. We will take individual differences in word-level reading behaviour as a starting point: as we 
will argue, these individual differences can likely be explained by both characteristics of the text as 
well as characteristics of the reader, and importantly, their mental simulation and experience of the 
story. We will then discuss how mental simulation might be reflected in reading behaviour and 
neural activity, and what evidence there is for individual differences in mental simulation on the 
behavioural and neural level. Bringing back the focus of the paper to the aforementioned individual 
differences in how people read, experience, and like stories, we will discuss some of the characteris- 
tics of readers, and what the role of simulation might (and might not) be in determining people’s 
reading preferences and experiences. Finally, we will suggest avenues for further research to eluci- 
date the cognitive and neural processes that underlie subjective reading experiences. We will limit 
ourselves mostly to the context of one common type of story, namely written, mostly literary, (short) 
stories and novels. 


Individual differences in reading 


A large proportion of research into how people process, experience, and like stories has been 
conducted in the context of reading. This is not entirely surprising given that, notwithstanding 
centuries of oral storytelling traditions as well as the more recent advent of audiobooks, much of our 
exposure to literary stories takes place through written books. Although we note that more and more 
studies also focus on the cognitive processing of stories via other media such as movies (e.g., Grall et 
al.; Milivojevic et al.) or comics (e.g., Cohn and Schilperoord), for the sake of this paper, we will 
mostly focus on written literature. 

Written literature is a highly suitable medium for studying cognitive processes during reading as 
it allows for the measurement of reading times and eye movements. A long tradition of research has 
been built on the idea that eye tracking can be used to measure attention and processing speed 
during reading (Rayner, “Eye Movements in Reading and Information Processing: 20 Years of 
Research.”; Rayner, “Eye Movements in Reading: Models and Data.”; Klieg] and Laubrock; Just and 
Carpenter). For instance, words or passages that require increased attention are associated with 
longer reading times. This relationship can be observed in word features such as lexical frequency 
and word length. Historically, it has been found that words that occur frequently in a language are 
easier to recognize and process, thus requiring less attention (Juhasz; Juhasz and Rayner; Rayner, 
“Eye Movements in Reading and Information Processing: 20 Years of Research.”; Rayner and 
Duffy). Although this word frequency effect has been replicated many times over the years 
(Kuperman et al.), it has been found that this effect is subject to individual differences, for example in 
language exposure (Brysbaert, Mandera, and Keuleers). Moreover, the word frequency effect has 
been found to be particularly visible in medium-frequency words as opposed to high-frequency 
(i.e., over-learned) or low-frequency (under-learned) words (Brysbaert, Mandera, and Keuleers). 
Similarly, word length has a strong effect on reading times: shorter words are easier to process and 
require less attention than longer words, as demonstrated by shorter gaze durations for these words 
(Rayner, “Eye Movements in Reading and Information Processing: 20 Years of Research.”). Recent 
work has argued that word length might even be a stronger predictor of measures of both early- and 
late-stage language processing than, for instance, word frequency (Kuperman et al.). Additionally, 
reading speed is not only affected by the characteristics of individual words, but also by the context 
in which they occur. Words that are highly predictable in their context are easier to process and 
therefore associated with shorter gaze durations (Goodkind and Bicknell; Hale; Levy; Lopopolo et 
al.; Kuperman et al.). 

These word characteristics, together with others such as word prevalence (i.e., how well-known a 
word is; Brysbaert, Mandera, McCormick, et al.), age of acquisition (Brysbaert, Buchmeéier, et al.), 
and similarity to other words (e.g., Brysbaert, Mandera, McCormick, et al.; Adelman and Brown) 


18 | JOURNAL OF COMPARATIVE LITERATURE AND AESTHETICS 


are strong predictors of word processing times, explaining almost 50-70% of variance in lexical 
decision times (i.e., deciding whether something is a real word; Brysbaert, Mandera, McCormick, et 
al.). These surface-level word characteristics are also important predictors of reading times in com- 
plex reading tasks such as poetry reading and literary reading (e.g., Xue et al.; Eekhof et al.) suggest- 
ing that they are robust indicators of cognitive processing across reading tasks. 

Given the strong predictive value of surface-level word characteristics, the text in terms of its 
words thus appears to pose strong constraints on reading behaviour. Indeed, the alignment between 
reading times and complexity (also known as cognitive coupling) is an important predictor of text 
comprehension, as stronger alignment is associated with better comprehension (Mills et al.; Rayner 
et al.). However, this also implies that the strength of this alignment varies among readers. Indeed, it 
logically follows from the findings cited above that roughly 30-50% of variance is unaccounted for 
by the combination of lexical factors studied so far! (Brysbaert, Mandera, McCormick, et al.), sug- 
gesting that the words in the text itself might not be the only predictor of word processing times. 

Indeed, another important predictor of word processing times is individual differences in the 
reader. Individual differences in reading skill have been shown to be predictive of low-level reading 
behaviour (Eekhof et al.). For instance, the degree to which reading times are predicted by word 
frequency depends on an individuals’ reading skills (e.g., Ashby et al.), vocabulary (Mainz et al.), 
educational background (Tainturier et al.), and print exposure (Chateau and Jared; Sears et al.; 
Eekhof et al.). Similar relationships have been shown for the degree to which other word character- 
istics influence reading times, such as how similar words are to other words, and how long they are 
(Barton et al.; Chateau and Jared; Sears et al.; Spinelli et al.). It has been proposed that these differ- 
ences between readers of different skill levels arise from differences in the level of automaticity in 
their word processing (Brysbaert, Mandera, and Keuleers). As readers are exposed to more words, 
they develop more stable lexical representations, which allow for faster word recognition. Less 
skilled readers therefore need to rely on word characteristics and word context more than skilled 
readers (Kuperman and Van Dyke; Perfetti et al.). 

More developed reading skills are also linked to making better or stronger predictions about 
which words are upcoming based on what is being read: in a sentence like “the brave knight saw the 
fierce dragon and reached for his ...”, most readers are likely to insert the word “sword” (example 
based on Otten and Van Berkum). The ability to accurately make such predictions allows for faster 
reading, and for skipping predictable words: around 25-33% of words in a text, and around 75% of 
the highly frequent two-letter words, are skipped by skilled readers (Leinenger and Rayner; Rayner 
and McConkie). Less skilled readers are likely to read more slowly (in terms of fixation durations), in 
particular when they encounter low-frequency words, and are less likely to skip short words (Ashby 
et al.; Haenggi and Perfetti; Leinenger and Rayner). Recent work has confirmed that word skipping 
might be a stable individual difference during literary reading, which might be relatively indepen- 
dent of what is being read and how the reader experiences the text (Faber, Mak, et al.). 

However, the experience of reading a literary story is more than mere word processing: from time 
to time, people also feel absorbed or “lost” ina story. Recent work has shown that people not only 
vary in their sensitivity to surface-level word characteristics during literary story reading, but that 
these variations are also related to how absorbed people are in the story and how much they like it 
(Eekhofet al.). Absorption is an experiential state? in which readers feel transported to the world of 
a story, experience emotional responses to the described characters and events, have a vivid mental 
image of what they read, and are generally focused on the story world is known as story world 
absorption (Kuijpers, Absorbing Stories: The Effects of Textual Devices on Absorption and Evaluative 
Responses). How strongly a story “grasps” a reader might vary from reader to reader and from story 
to story (Gerrig; Green and Brock; Jacobs and Willems; Kuijpers et al.; Kuzmicova), as absorption is 
inherently a subjective phenomenon. Specifically, a higher degree of narrative absorption has been 
linked to lower degrees of sensitivity to word length, and liking a story more is associated with 
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decreased sensitivity to word frequency, suggesting that being somewhat “detached” from the story 
in terms of word characteristics is associated with a more pleasant and engaging reading experience 


(Eekhof et al.). 


How mental simulation influences reading 


So far, we have argued that there might be a link between word characteristics, which influence 
reading on the lexical level, and reading experiences such as narrative absorption, which arises at the 
level of the story (or at least at the level of plot events; cf. Pianzola et al.), and is likely to influence 
reading both at higher and lower levels. An interesting phenomenon that might be one of the drivers 
of this link is mental simulation. In the context of language comprehension, an often-used definition 
of mental simulation is “the re-enactment of perceptual, motor, and introspective states acquired 
during experience with the world, body, and mind” (Barsalou, p. 618), which might occur when 
people encounter language that refers to these states. A word like “kick”, for instance, can automati- 
cally elicit motor simulation, whereas a description of a visual quality of an object can elicit percep- 
tual simulation (Pulvermiiller et al.; Moody and Gennari). The simulation of introspective states, for 
instance what a character ina story is thinking or feeling, is also known as mentalizing (Goldman, 
“Theory of Mind”). 

Evidence for the role of mental simulation in language processing stems from the embodied 
cognition literature, which has proposed that language comprehension recruits mental simulations 
(although it has been debated to what degree this is the case; Muraki et al.). Tasks like the sentence- 
picture verification task (Stanfield and Zwaan) and the action-sentence compatibility effect (Glenberg 
and Kaschak) have been used as evidence that language processing does indeed rely on mental 
simulation: for instance, evidence has suggested that when presented with a picture of either a 
horizontally or vertically oriented picture of a pencil after hearing the sentence “John put the pencil 
in the cup”, people find it easier to verify that a pencil was mentioned when they are presented with 
the congruent visual presentation (i.e., the vertical pencil) than the incongruent one (Stanfield and 
Zwaan). Similarly, after reading the sentence “Close the drawer”, responses have been shown to be 
faster when they are in line with the direction of the movement (in this case, away from the body; 
Glenberg and Kaschak). 

The findings described above, together with many others, have led to the idea that language must 
at least to some extent be embodied. However, the degree to which language is embodied has been 
a topic of much debate (see, e.g., Muraki et al.). For instance, the action-language compatibility 
effect has been found to be difficult to replicate (Morey et al.; Winter et al.), and not all effects found 
in sentence-picture verification tasks have been found to be equally strong (Zwaan and Pecher). 
This has led researchers to conclude that strong embodied theories, in which all concept knowledge 
is solely represented in simulations, are difficult to substantiate (Muraki et al.). At the same time, 
consistent replications of colour and shape effects in sentence-picture verification tasks make it 
difficult to substantiate radically unembodied theories of language processing (in which mental 
simulations do not play a role in language processing whatsoever; Muraki et al.). Importantly, these 
findings combined are suggestive of some role of mental simulation in language processing, although 
it is not clear to what extent simulation is necessary or sufficient (for a review see Muraki et al.). 

As recently reviewed by Muraki and colleagues, in the weakest embodied view, meaning is stored 
in amodal, symbolic representations that can be enriched with associated sensory information (Mahon 
and Caramazza). Slightly stronger but still weakly embodied views go a bit further in stating that 
sensory information in part constitutes the meaning of words, together with linguistic information 
(for a recent overview, see Muraki et al.). Without committing to a specific account of embodied 
cognition, a commonality across these theories is that each would predict that language processing 
might elicit the activation of (memory traces associated with) emotional, perceptual, social, and 
sensorimotor processes. Based on these commonalities, we will use the following working definition 
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of simulation during language comprehension here: we regard simulation as the automatic (re-)acti- 
vation of sensory and emotional information during language processing (Mak and Willems, “Men- 
tal Simulation during Literary Reading”). 

Providing evidence for a role of simulation in language processing, recent work has shown that 
mental simulation might be one of the factors driving individual differences in reading on the lexical 
level. Mak and Willems (Mak and Willems, “Mental Simulation during Literary Reading: Individual 
Differences Revealed with Eye-Tracking”) used eye tracking to study individual differences in 
mental simulation (as defined above) during literary short story reading. One group of 90 partici- 
pants read three literary short stories, and underlined all words, sentences, and passages that they 
considered to be motor descriptions, perceptual descriptions, and descriptions of mental events, 
leading to an average “simulation score” for each category for each word. These descriptions were 
hypothesised to afford elicitation of automatic activation of sensory and emotional information (or 
memory traces) stored along with the described concepts or situations. Descriptions that were un- 
derlined by more participants were considered to be more likely to (re-)activate such information. 
A second group of 109 participants then read each story, after which the simulation scores (and 
several important lexical characteristics) were regressed against their eye movements. The results 
revealed that while reading motor descriptions—presumably eliciting more motoric simulation— 
readers sped up, whereas during perceptual and mentalizing descriptions, readers slowed down. 
However, there were striking individual differences in these relationships: while readers on average 
slowed down during mentalizing, some participants sped up, suggesting that the degree to which 
readers simulate (or the nature of their simulation) might vary. 

Importantly, this implies that there might be variation in the degree of simulation as revealed by 
readers’ eye movements: some readers showed strong evidence for simulation (and did for all kinds 
of simulation studied), whereas other showed only weak evidence or no evidence at all for any of the 
kinds of simulation studied. Moreover, these individual differences in simulation were linked to 
absorption and story liking: for instance, people who experienced a stronger emotional response to 
the story had a stronger effect of mental simulation on their gaze behaviour. In addition, people who 
focused more on the story (which is an aspect of absorbed reading) displayed a weaker association 
between motor simulation and mentalizing on the one hand and gaze duration on the other hand. 
These findings suggest that indeed, mental simulation might be a factor that explains the observed 
link between reading at the lexical level and people’s reading experience in terms of absorption. 

Recent work has replicated these findings in a study that combined eye tracking with functional 
magnetic resonance imaging (fMRI) (Mak, Faber, et al., “Different Kinds of Simulation during 
Literary Reading: Insights from a Combined FMRI and Eye-Tracking Study”). Motor simulation 
was again linked to faster reading, and perceptual simulation and mentalizing to slower reading, 
with similar effect sizes as reported in the previous study, suggesting that these findings are robust 
across experimental settings, and that they are likely to be indicative of robust effects of simulation 
on mechanistic reading. Other recent work (Magyari et al.) has shown that the number of adjectives 
and verbs—which might be linked to simulation by providing (detail to) perceptual, motoric, or 
mentalizing descriptions that elicit simulation—on a page is also predictive of reading times: more 
adjectives lead to longer reading times (slower reading), whereas more verbs lead to shorter reading 
times (faster reading). These findings suggest that the use of different word categories might provide 
a potentially useful handle on manipulating simulation across texts. 

In addition to variation across readers, the stories being read also impact the degree to which 
people mentally simulate during literary story reading. In stories that describe the interaction of 
characters with their environment as opposed to describing the environment without the interac- 
tion with a character, fixation durations are longer for stories that described interactions between 
characters and their environment, suggesting increased processing and possibly higher degrees of 
simulation (Magyari et al.). 
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The neural basis of mental simulation during story reading 


As argued above, evidence suggests that individuals might vary in the degree to which mental 
simulation occurs during story reading. This prompts the question of what the neural basis of mental 
simulation during story reading is, and whether there is any evidence to suggest that neural activity 
associated with simulation is linked to how people subjectively experience stories. As we will discuss 
below, much of the “traditional” embodied language processing literature has focused on evidence 
for domain-specific simulation (e.g., Hauk et al.). However, more recently, the focus has shifted to 
identifying commonalities across different cognitive processes (e.g., Addis, “Are Episodic Memories 
Special? On the Sameness of Remembered and Imagined Event Simulation”). In our discussion here, 
we will follow this distinction: we will first discuss evidence for domain-general processes linked to 
simulation, followed by a discussion of domain-specific processes. 

Outside of the literature on language processing, simulation has been defined more broadly as “the 
mental rendering of experience” (Addis, “Mental Time Travel? A Neurocognitive Model of Event 
Simulation”, p. 234; note that this definition is much broader than our working definition, which 
only captures the automatic activation of sensory and emotional information during language pro- 
cessing). Addis (Addis, “Are Episodic Memories Special? On the Sameness of Remembered and 
Imagined Event Simulation”; Addis, “Mental Time Travel? A Neurocognitive Model of Event 
Simulation”) argues that simulation is a domain-general process, which is closely related to (or even 
overlaps with) semantic and episodic memory, mental time travel (remembering the past and imag- 
ining the future), counterfactual thinking, creativity, theory of mind, narrative comprehension, and 
event perception (Addis, “Mental Time Travel? A Neurocognitive Model of Event Simulation”). 
These processes rely on internally constructed representations (rather than externally driven ones), a 
process typically associated with activity in the Default Mode Network (DMN) of the brain (Buckner 
and DiNicola). 

Although studies focusing on mental simulation in the (narrow) sense of our working definition in 
the context of story reading have been relatively scarce (see below), there is evidence to suggest that 
the DMN is involved in more general simulation and related processes such as event cognition 
during narrative processing as well. Striking evidence comes from a study that focused on neural 
synchrony across participants who had one of two possible interpretations of a story (i.e., the main 
character’s wife is cheating on him, or he is paranoid) (Yeshurun et al.). Participants who had the 
same interpretation displayed greater synchrony across the group within the DMN during listening 
(Yeshurun etal.), suggesting that the DMN might be involved in processes that strongly relate to 
mentalizing (ie., using inferences about the mental state of a character to interpret the story). Such 
synchronies across individuals during story processing have also been observed in movie watching 
(Chen et al.), where accessibility to prior information determines how similar neural activity in 
DMN areas is across participants. For example, when comparing two groups watching the second 
part of a movie that critically relies on information from the first part of the movie that one group 
watched immediately prior and the other group one day before, activity in the DMN during the 
first minutes of the second part of the movie is asynchronous between groups, but synchronous within 
groups, and increases in synchrony over the course of the movie (Chen et al.). Additionally, when 
participants read a story with alternating, independent storylines, they showed storyline-specific 
neural patterns, particularly in the DMN (Chang et al.). Memory performance for the story turned 
out to be best in those participants in which this activation was most pronounced (Chang et al.). 

Within the DMN, a special role seems to be reserved for the angular gyrus (Addis, “Mental Time 
Travel? A Neurocognitive Model of Event Simulation”). Addis (Addis, “Mental Time Travel? A 
Neurocognitive Model of Event Simulation”) proposes that activity in higher-level, posterior DMN 
regions, such as the angular gyrus, is non-linearly related to the associative strength between the 
elements of simulations: simulations comprised of information from multiple modalities or higher- 
order processing regions will often be more coherent than simulations comprised of information 
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from one sensory modality, and will therefore have more associative strength, and place fewer 
constructive demands on the posterior regions of the DMN. In contrast, simulations with lower 
associative strength place higher demands on the DMN, resulting in stronger activation of these 
regions (Addis, “Mental Time Travel? A Neurocognitive Model of Event Simulation”). However, if 
too much demand is placed on these posterior regions of the DMN (when associative strength is too 
low), the activity in these regions diminishes at which point activity in anterior DMN and fronto- 
parietal network increases (Addis, “Mental Time Travel? A Neurocognitive Model of Event Simu- 
lation”). The angular gyrus, together with the adjacent supramarginal gyrus, has indeed been found 
to play a role in many of the cognitive processes that are important for simulation, such as stimulus- 
driven attention, social cognition, episodic memory, and self-generated, spontaneous thought 
(Igelstrém and Graziano), literary reading (Hartung, Wang, et al.), referential indexing (Matchin et 
al.), and situation model updating (with mental simulation playing an important role in building and 
updating a mental representation (or situation model) of a story; Zwaan and Radvansky; Kurby and 
Zacks, “The Activation of Modality-Specific Representations during Discourse Processing”; Zwaan). 
The angular gyrus also plays an important role in event segmentation across domains: a posterior- 
to-anterior hierarchy in the perception of event boundaries has been found in studies of listening to 
audio narratives, movie viewing, and music listening (Baldassano et al.; Williams et al.). Similarly, 
the supramarginal and angular gyri have been hierarchically associated with anticipation of event 
boundaries in repeated movie viewing (Lee et al.). The supramarginal and angular gyri modulate 
functional connectivity between domain-specific language areas and reading related domain-gen- 
eral executive processing regions in the prefrontal cortex (Kim et al.). 

Taking together this evidence, the angular/supramarginal gyrus is a good candidate for a do- 
main-general simulation area, which has been supported by neural evidence from Mak, Faber, and 
Willems (Mak, Faber, et al., “Different Kinds of Simulation during Literary Reading: Insights from 
a Combined FMRI and Eye-Tracking Study”). In a study that used simultaneous fMRI with eye 
tracking, Mak, Faber, and Willems asked forty participants to read two literary short stories (coded 
for motor descriptions, perceptual descriptions, and descriptions of mental events; Mak and Willems, 
“Mental Simulation during Literary Reading: Individual Differences Revealed with Eye-Track- 
ing”) in the {MRI scanner. Subsequently, the participants answered questions about their experience 
and comprehension of the stories, and about their personal characteristics (empathy, transportabil- 
ity) and reading habits in daily life. As mentioned above, this study replicated the associations 
between eye movements and words that elicit simulation that were observed in previous studies. 
The open question, however, was whether domain-general simulation areas could be discovered 
and/or whether there are modality-specific brain activation patterns that are associated with the 
different types of simulation (motor, perceptual, and mentalizing). Perhaps unsurprisingly, Mak, 
Faber, and Willems (Mak, Faber, et al., “Different Kinds of Simulation during Literary Reading: 
Insights from a Combined FMRI and Eye-Tracking Study”) found evidence for a domain-general 
simulation area in the left supramarginal gyrus, in keeping with the existing theories and empirical 
evidence suggesting a role for this area in domain-general simulation. 

In addition, Mak, Faber, and Willems also found some evidence for domain-specific simulation 
areas, in line with previous evidence for domain-specific brain activity associated with motor 
simulation (N ijhof and Willems; Chow et al.; Kurby and Zacks, “The Activation of Modality- 
Specific Representations during Discourse Processing”), visual simulation (Chow et al.; Tamir et 
al.), auditory simulation (Kurby and Zacks, “The Activation of Modality-Specific Representations 
during Discourse Processing”), and mentalizing (Nijhof and Willems; Tamir et al.). 

Specifically, the results from Mak, Faber, and Willems (Mak, Faber, et al., “Different Kinds of 
Simulation during Literary Reading: Insights from a Combined FMRI and Eye-Tracking Study”) 
revealed that motor simulation was associated with activity in brain areas that have previously been 
linked to processing motor verbs (e.g., Kurby and Zacks, “The Activation of Modality-Specific 
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Representations during Discourse Processing”; Moody and Gennari; Nijhofand Willems), such as 
the cingulate and paracingulate cortex, precuneus, parahippocampal gyrus, and middle and superior 
frontal gyrus. Furthermore, brain areas associated with inferencing, event segmentation, and situa- 
tion model building, such as angular gyrus, subcallosal cortex, and frontal medial cortex (e.g., Kurby 
and Zacks, “Segmentation in the Perception and Memory of Events”; Speer, Zacks, et al.; Speer, 
Reynolds, et al.) were also activated when people read motor descriptions. Note that adjacent areas 
in these same networks were activated by domain-general mental simulation. This might indicate a 
special role for motor simulation (or action processing) in mental simulation in general. 

One possible (but speculative) explanation for this finding would be that actions have been found 
to be drivers of event segmentation and hence, situation model building, as actions are often associ- 
ated with changes in agents and objects (and their interactions), the intentions of the protagonist, 
causal relationships in a story, and shifts in space and time (Faber and Gennari; Faber, Radvansky, et 
al.; Zwaan and Radvansky; Kurby and Zacks, “Segmentation in the Perception and Memory of 
Events”). This could for example be studied by asking participants to segment passages, while mo- 
dality-specific motor simulation areas are suppressed (for example with rTMS). If event segmenta- 
tion is impaired while motor areas are suppressed (but not when motor areas are not suppressed), this 
can tell us more about the necessity of motor simulation for event segmentation. The potential 
necessity of motor simulation for these processes, however, does not explain why people speed up 
when they read motor descriptions, as people normally slow down when they need to update their 
situation model (Pettijohn and Radvansky). Although these findings are in line with previous research 
showing that more action-laden sentences are processed faster (Marino et al.), more research is needed 
to find out why motor simulation is processed at a higher speed than other kinds of simulation. 

Perceptual simulation elicited activity in several modality-specific areas that were also observed 
in previous work, such as fusiform and parahippocampal gyrus, and inferior temporal cortex. Pre- 
vious work has shown that these higher- and lower-level visual areas are involved in visual simula- 
tion (Chow et al.; Tamir et al.). These areas, together with others, are important components of the 
ventral visual pathway, which processes objects, faces, and scenes (Grill-Spector and Weiner; Kanwisher, 
Kravitz et al.), suggesting that perceptual simulation elicits domain-specific activation in the brain. 

Mentalizing was found to elicit activity in areas that have previously been associated with 
mentalizing in general: regions in the temporal pole, parietal operculum, anterior cingulate, and 
angular gyrus (Frith and Frith; Igelstr6m and Graziano; Laurita et al.; Paulus et al.; Saxe and Kanwisher). 
Apart from that, mentalizing was found to elicit activity in the reading network in the brain (e.g., 
middle temporal gyrus, inferior frontal gyrus, superior temporal gyrus, planum temporale, cerebel- 
lum). Associations between mentalizing and language processing have been found before, particu- 
larly in the context of social cognition and Theory of Mind (Tamir et al.; Hertrich et al.). The ability 
to understand and make inferences about the intentions of others relies on the Theory of Mind 
system, which is neuroanatomically distinct from the language network, but has been shown to co- 
activate during language comprehension (Paunov et al.; Hertrich et al.). This coordination between 
networks suggests that language might be important for our ability to understand others, and that 
our ability to understand language might (to some extent) be dependent on our social skills, suggest- 
ing that these two abilities might be closely intertwined. Indeed, recent evidence shows that perfor- 
mance on Theory of Mind-tasks is dependent on linguistic experience, where children with less 
linguistic experience due to later-than-normal first linguistic exposure perform worse on Theory of 
Mind tasks than their peers with more linguistic experience (and comparable to younger children 
with similar linguistic experience; Richardson et al.). Moreover, development of Theory of Mind (but 
also other cognitive functions) is proposed to be dependent on linguistic development (de Villiers). 

Note that there were also passages in the stories that elicited more than one form of simulation (for 
example both motor simulation and mentalizing or motor simulation and perceptual simulation). In 
these instances, modality-specific activation could still be observed, but from different modalities 
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simultaneously. In addition, there are of course other types of descriptions that afford mental simu- 
lation that were not explicitly studied here: physical motion of characters in stories has been shown 
to elicit activation in the posterior temporal cortex/angular gyrus (i.e., a region implicated in the 
perception of biological motion), dialogue in stories is associated with activity in many regions in 
the bilateral temporal and inferior frontal cortices and the temporo-parietal junction, and the iden- 
tities of different story characters have been shown to activate the right posterior superior/middle 
temporal region (Wehbe et al.). It is likely—although to our knowledge not empirically validated to 
date—that these types of simulation also vary across individuals, and might play a role in subjective 
experiences during reading. 


Different reader types? 


As explained above, Mak, Faber, and Willems (Mak, Faber, et al., “Different Kinds of Simulation 
during Literary Reading: Insights from a Combined FMRI and Eye-Tracking Study”) discovered 
domain-specific and domain-general areas associated with (different kinds of) mental simulation. In 
addition, they found evidence for individual differences in mental simulation: how strongly brain 
areas responded to simulation-eliciting content (in terms of percent signal change) was related to 
measures of story appreciation and trait-level, personal characteristics. Specifically, people whose 
brains showed stronger responses to motor descriptions on average thought that the stories were 
more beautiful, witty, or special, and scored higher on the Fantasy subscale of the Interpersonal 
Reactivity Index (IRI; Davis). Similarly, people who were more sensitive to perceptual descriptions 
in terms of percent signal change in the brain scored higher on the Fantasy subscale, and scored 
higher on the Perspective Taking subscale of the IRI. Being more sensitive to mental event descrip- 
tions was related to finding a story more suspenseful, interesting, captivating, or gripping. Taken 
together, these findings suggest that how mentally “involved” people are in a story is reflected in 
their brain activity. 

Contrary to their expectations, Mak, Faber, and Willems did not observe a relationship between 
the strength of the neural responses to simulation words and story world absorption. Although 
contrary to expectations, this finding may not be entirely surprising. For example, engagement with 
narrative movies was not found to be correlated with individual differences in time perception, due 
to viewers becoming more similar (individual differences decrease) with increased engagement 
(Cohen et al.). These findings have been shown to extend to spoken political speeches, where neural 
synchrony in language areas during processing is higher for rhetorically strong than rhetorically 
weak speeches (Schmilzle et al.). Although previous studies have pointed out that findings done in 
the context of narrative film do not necessarily translate to reading (Loschky et al.; Hutson et al.; 
Hubbell et al.), ifindividual difference findings from narrative movie processing do translate to the 
context of mental simulation during story reading, results might be most apparent when looking at 
individual differences between participants reporting low absorption scores, for example using 
inter-subject representational similarity analysis (Finn et al.). 

Alternatively, the lack of a relationship between the strength of the neural response to simulation 
words and story world absorption as found by Mak, Faber, and Willems could be due to the fact that 
simulation is not an important factor driving absorption, but it might also indicate that reading on 
the neural level is better predicted by more stable, trait-level differences rather than experiential 
states. This is in line with multiple previous studies that found that trait-level individual differences 
were more strongly associated with simulation than readers’ level of absorption in response to the 
stories they read (e.g., Faber, Mak, et al.; Hartung, Hagoort, et al.; Hartung, Wang, et al.; Mak, De 
Vries, et al.; Van den Hoven et al.). These findings suggest that although simulation might still play 
a role in absorption (Kuijpers, “Bodily Involvement in Readers’ Online Book Reviews: Applying 
Text World Theory to Examine Absorption in Unprompted Reader Response”), individual varia- 
tion in simulation is not directly predictive of individual variation in story world absorption. 
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As mentioned above, relatively stable, trait-level characteristics such as what kind of stories people 
like and their score on the interpersonal reactivity index are linked to how the brain responds to 
stories, with specific types of simulation being linked to specific patterns of preferences and inter- 
personal reactivity. This resonates with previous work that has shown that there are different “fin- 
gerprints” of aesthetic appreciation (Mak, Faber, et al., “Different Routes to Liking: How Readers 
Arrive at Narrative Evaluations”). Overall, people’s interest in a story, how suspenseful it is, how 
amusing it is, and how beautiful people think it is were all positively associated with how much 
people like a story. However, people vary in the extent to which each of these components contrib- 
utes to their assessment: some people for instance appreciate sadness in stories, whereas others do not. 
Such individual differences might be linked to the processing of emotional information on the 
neural level, as also suggested above. Readers might therefore—although the causal direction is 
unknown—have different preferences, and different reading styles. 

Indeed, a survey of 501 fiction readers in The Netherlands has found that different readers take on 
different roles as a reader and have different expectations of the reading experience (Riddell and van 
Dalen-Oskam). Although the survey revealed no strict boundaries between different types of readers, 
it shhowed that they do vary in terms of how they engage with fiction: some readers are predominantly 
“identifying” readers who enjoy fiction that allows them to empathise with the main character, 
whereas more “distanced” readers predominantly enjoy aesthetic and stylistic aspects of literature. 
However, some readers characterise themselves as both identifying and distanced readers. The 
survey revealed that readers are perhaps best differentiated in terms of how many different experiences 
they expect during the reading process: some readers expect mostly to be transported by the narrative 
(being “swept away” by the story, Riddell and van Dalen-Oskam, p. 10), whereas others expect a 
story to also be intellectually challenging. Riddell and Van Dalen-Oskam speculate that the latter 
category might possess a number of different “reading techniques”, such that they can choose how to 
approach a literary story and thereby experience it in different ways and/or on different levels. 
Despite this being a speculative account, it opens up the possibility that the extent to which an 
individual has access to an arsenal of different reading techniques is an important factor in determining 
how people experience and enjoy stories. 

This idea is supported by the Neurocognitive Poetics Model of Literary Reading (Jacobs), which 
has proposed a distinction between two routes of literary reading: a fast route and a slow route. The 
fast route is evoked by reading familiar, high-frequency, and highly imageable words, leading to a 
fluent reading experience that makes a reader feel immersed in the story. As highly imageable words 
might also lead to higher degrees of simulation, the (subconscious and automatic) process of simula- 
tion might play an important role in evoking this reading experience (Mak and Willems, “Mental 
Simulation during Literary Reading”). This route is also known as the affective processing route, as 
people experience “fiction feelings” —feelings such as transportation, emotional engagement, and 
empathy. The slow route on the other hand is triggered by foregrounded elements —stylistic 
devices such as metaphors, abstract, rhythmic and rhetorical structure, and defamiliarizing lan- 
guage, evoking a disfluent reading experience, but also evoking aesthetic feelings. The latter is 
thought to lead to higher aesthetic appreciation of literature and poetry. In line with the observa- 
tions made by Riddell and Van Dalen-Oskam (Riddell and van Dalen-Oskam), an open question is 
whether readers vary in the extent to which they can (intentionally or automatically) employ each 
route during reading. As such, whether or not readers appreciate a story and feel immersed might be 
an interaction between the reader and the text: the reading techniques that a reader possesses might 
determine their story preferences, and texts that appeal to techniques that a reader has limited access 
to might not be appreciated. 

There is some neural evidence to support the idea that readers vary in how they approach a story. 
Nijhofand Willems (Nijhofand Willems) found that some readers (or in the case of this fMRI study, 
listeners) might focus mostly on concrete events and descriptions of actions, whereas others mostly 
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attend to descriptions of thoughts, emotions and beliefs (i.e., mentalizing). These differences were 
reflected in the relative strength of each type of mental simulation in the brain: people who focused 
more on action content showed stronger responses in the left and right motor regions of the brain, 
whereas people who were more oriented towards mentalizing showed stronger responses in 
mentalizing regions such as the temporo-parietal junction, precuneus, and medial prefrontal cortex. 
It is important to note, however, that participants were not bimodally distributed, but rather formed 
a gradual “simulation spectrum” ranging from more action-oriented to more mentalizing-oriented. 
Although these findings do not directly map onto the distinction between the “fast” and “slow” 
processing route, they do lend support to the idea that there might be quantifiable differences in how 
individuals approach and/or process stories that are reflected in brain activity, providing insight 
into the neural underpinnings of how people experience stories. 


Discussion 


In this paper, we have discussed work that has studied how individuals vary in how they read, 
experience, and appreciate literary stories, and what that can tell us about the human mind. In 
particular, we have taken mechanistic reading behaviour as a starting point, and have argued that 
the way in which people read is linked to how people experience stories: being somewhat “detached” 
from the story is linked to stronger feelings of absorption. However, how people read might be more 
strongly linked to their reading abilities than it is to their current experience, and similarly, their 
reading preferences might be best predicted by the reading strategies or reading “modes/roles” that 
they have access to. This idea is supported by neural data that shows that people’s preferences and 
personal characteristics such as how they engage with literature is linked to the strength of their 
neural response to words that elicit simulation, an automatic and involuntary cognitive process. 
How people experience and like stories might therefore be best predicted by a combination of an 
individuals’ reading abilities, their propensity for simulation (which in turn might be linked to their 
reading abilities), and personal characteristics such as interpersonal reactivity, and the story that is 
being read. 

Arguably, in defining mental simulation as “the automatic (re-)activation of sensory and emo- 
tional information during language processing”, we have opted for a relatively “narrow” definition 
of simulation in comparison to colleagues studying, for example, the relationship between reading 
and empathy (e.g., Mar and Oatley; Bal and Veltkamp). In this area of research, stories themselves are 
seen as potential simulations of social situations, through which readers can practice their social skills. 
Historically, the term mental simulation has its origins in Simulation Theory, a theory in the phi- 
losophy of mind which describes how people understand the mental states of others (e.g., Goldman, 
Simulating Minds: The Philosophy, Psychology, and Neuroscience of Mindreading). According to Shanton 
and Goldman, in mental simulation “one mental event, state or process is the re-experience of 
another mental event, state, or process” (Shanton and Goldman, p. 528). The ‘re-experience’ in this 
description resonates with the definition of simulation by Barsalou (p. 618; “Simulation is the re- 
enactment of perceptual, motor, and introspective states acquired during experience with the world, 
body, and mind”) as well as our proposed working definition of mental simulation. However, we do 
acknowledge that our working definition of mental simulation, although grounded in established 
research traditions within Philosophy and Psychology, is fairly narrow (as is the scope of this paper) 
and would perhaps not be sufficient to capture phenomena studied in different fields of research. 

Of course, there are other important processes at work when going from single words to represen- 
tations of sentences and larger pieces of discourse such as stories (see, e.g., Kendeou and O’Brien), 
some of which might border on simulation-related processes. For instance, people vary in how 
associated knowledge is activated and integrated during reading (e.g., Kintsch, “The Role of Knowl- 
edge in Discourse Comprehension: A Construction-Integration Model.”; Kintsch, Comprehension. 
A Paradigm for Cognition). Recent theories argue that this knowledge is passively activated through 
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a process called resonance (Myers and O’Brien; Sonia and O’Brien), and is then integrated and 
validated. These processes play an important role in narrative reading, as they determine how for 
instance world knowledge is used to validate the consistency of a story (Cook and O’Brien). Impor- 
tantly, they might also to some extent pose a limitation on how much transportation can take place, 
as readers cannot “switch off” their own (world) knowledge when taking the perspective of the 
protagonist (Creer et al.). Although theories of resonance are beyond the scope of the present paper, 
they do provide an important avenue for future research, as it is currently an open question to what 
extent these processes overlap with people’s propensity to mentally simulate language. 

One important limitation of almost all of the studies cited here is that they rely on naturalistic 
variation in and across stories, without manipulations of variables of interest. In order to make causal 
claims about how individuals process specific narrative information, it might be necessary to con- 
duct more controlled experiments that manipulate specific text characteristics, such as the amount of 
simulation-inducing content without changing other aspects of the text. Another important limita- 
tion of studying mental simulation in the context of naturalistic variation in and across stories, is that 
most studies reported in this paper only use a limited number of stories. Since naturalistic narratives 
are rather long, the experiment would simply take too much of participants’ time if more than a few 
stories would be used in each experiment. In the case of Mak, Faber, and Willems (Mak, Faber, et al., 
“Different Kinds of Simulation during Literary Reading: Insights from a Combined FMRI and Eye- 
Tracking Study”), for example, only two stories were used in the study of individual differences in 
simulation in the brain. This makes it difficult (if not impossible) to study story differences in most 
studies using naturalistic stories, and future studies with different stories would be necessary to 
determine whether the found effects also translate to other stories. That said, Mak, Faber, and Willems 
looked specifically at words and passages in stories, rather than at entire stories. Although this cannot 
completely account for possible differences between stories (either in plot or in language use), it does 
make that the number of observations on which their conclusions are based is larger than two (also note 
that their eye-tracking findings show some differences between stories, but they are slight differences in 
the strength of the found effect, and not in the direction of the effect). 

Despite the abovementioned disadvantages, the use of naturalistic narratives does also have im- 
portant advantages: results are high in ecological validity, and relative contributions of different 
variables can easily be assessed due to their natural occurrence (Willems et al.). A mixed approach 
that makes small changes to existing, naturalistic narratives might therefore be an important way 
forward to obtaining new insights into the relationship between readers, their reading experience 
and appreciation, and the stories that they read. 

In this paper, we have mostly focused on the cognitive process of mental simulation as a window 
into people’s reading experience. However, as we have argued above, simulation is only one aspect 
of language processing, and might in and of itself only be weakly predictive of people’s mental state. 
Indeed, loosely based on Jacob’s Neurocognitive Poetics Model of Literary Reading (Jacobs), read- 
ing experiences (or ‘fiction feelings’) can be defined as absorption, transportation, emotional en- 
gagement, empathy, story enjoyment, and reading fluency, among others. As can be gleaned from 
the literature discussed here, most neuroscientific research has focused on one of these aspects, rather 
than on their constellation. These factors are likely to influence how people appreciate literature: 
behavioural research has for instance shown that people like a poem better when it is presented in an 
easy-to-read font, suggesting that reading fluency is important for appreciation (Gao et al.). An 
open challenge for neuroscientific research is therefore to assess the contributions of and interplay 
between different subjective reading experiences, how they are represented or “implemented” in 
the individual reader, and how they are linked to people’s reading preferences. 

As mentioned above, the observation that readers whose reading is more “detached” from the 
actual text on average experience higher degrees of absorption, is an interesting observation, as 
detachment is often regarded as a sign of disengagement (e.g,, Mills et al.), and has been linked to the 
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process of “mindless reading” or “mind wandering” (Faber, Krasich, et al.; Faber and D’Mello; Faber, 
Bixler, et al.; Dias da Silva, Postma, et al.; Dias da Silva, Faber, et al.). Mind wandering is often 
defined as a state in which people’s attention shifts from the (reading) task to self-generated, task- 
unrelated, stimulus-independent thoughts (Smallwood and Schooler), and is linked to activity in the 
default network of the brain (DMN; e.g., Christoff et al.). However, previous work has shown that 
approximately 50% of mind wandering thoughts during reading or movie watching are directly 
triggered by what people are reading or watching (Faber and D’Mello), suggesting that some degree 
of engagement with a text might be necessary to instigate those mind wandering episodes. Indeed, 
recent theoretical work has taken the stance that mind wandering might be a “by-product” of task- 
relevant, attention driven processes, such as the construction of situation models during reading 
(Fabry and Kukkonen). 

Indeed, in the study by Mak and Willems (Mak and Willems, “Mental Simulation during Literary 
Reading: Individual Differences Revealed with Eye-Tracking”) described above, it appeared that 
people who focused more on the story (which is an aspect of absorbed reading) displayed a weaker 
association between motor simulation and mentalizing on the one hand and gaze duration on the 
other hand. The authors interpreted this finding as being somewhat reminiscent of “mindless read- 
ing”. While the theories proposing an association between attention, absorption, and mind-wan- 
dering await further empirical validation, we would like to add to this challenge that paradoxically, 
mind wandering and story world absorption might share cognitive and neurobiological underpin- 
nings: both rely on or cause a certain degree of perceptual decoupling (i.e., the detachment from the 
external world, and shifted focus toward the internal world), which is linked to activity in the default 
network of the brain that includes the domain-general simulation area that Mak, Faber, and Willems 
(Mak, Faber, et al., “Different Kinds of Simulation during Literary Reading: Insights from a Com- 
bined FMRI and Eye-Tracking Study”) observed in the inferior parietal lobe (the supramarginal 
gyrus). The default network of the brain has additionally been found to play a role in counterfactual 
thinking (see De Brigard and Parikh): imagining oneself outside of the here and now. Taken to- 
gether, the results from these different areas of research imply that the DMN is indeed involved in 
simulation, since it is activated whenever people are mentally simulating themselves in a different 
world (i.e., a story world, or a world outside of the here and now). 

Above, we discussed several of the individual-level characteristics that influence mechanistic 
reading—and thereby, the subjective reading experience—such as print exposure, vocabulary size, 
and education level. However, there are other factors that vary between readers, such as people’s 
own experience. If we take an embodied perspective on reading (i.e., meaning is to some extent 
grounded in people’s own [bodily] experiences), then people’s physical (and mental) experience and 
expertise should be predictive of their reading experiences. Indeed, research has shown that for 
instance expert volleyball players more strongly simulate action verbs depending on whether the 
words refer to the domain of their motor expertise (Tomasino et al.). Differences in (embodied) 
representation across individuals (and potentially cultures) might therefore play an important role in 
how individual readers experience stories. 

In this paper, we set out to discuss how individual differences in story processing, experience, and 
liking can inform us about the human mind and brain. We have discussed individual differences in 
reading behaviour, both in sensitivity to certain word characteristics and in subjective experiences 
such as story appreciation. Although relatively little research has studied the link between mental 
simulation and reading experiences using neuroimaging techniques, there is some evidence to sug- 
gest that people vary in the degree to which they mentally simulate words, and that these differences 
are reflected in neural activity in both domain-specific and domain-general areas of the brain. 
However, as we discussed, mental simulation by itselfis only a weak predictor of state-level subjec- 
tive reading experiences. In future research, it is important to therefore move away from mental 
simulation in isolation, and instead focus on the constellation of factors that contribute to subjective 
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experiences during reading. Fruitful steps have recently been taken in this direction, for example in 
studies looking at the experience of “flow” and mind wandering during reading. Only when inves- 
tigating these kinds of experiences further, not only with paper and pencil tasks or questionnaires, but 
for example also with eye tracking or neuroimaging, will we uncover what makes readers so different 
from each other and what factors contribute to the uniquely human experience of enjoying a story. 
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Notes 


"Note that, although many lexical word characteristics can be distinguished, they do not all equally influence 
reading behaviour (Xue et al.). A possible explanation could be that some are highly correlated, taking one 
into account may limit the additional explanatory power of another. Although it is impossible to state with 
certainty that exactly 30-50% of variance is unaccounted for by any lexical factor at all, we can be fairly 
certain that a significant portion of the variance in lexical decision times is unaccounted for by lexical factors. 

? We are talking about an experiential state, however, this does not mean that subjective experiences during 
story reading remain at the same level throughout a story reading experience. Indeed, just as word processing 
times vary between words depending on word characteristics (as explained above), absorption in stories has 
been proposed to vary throughout the story, depending on plot events or language errors, for example 
(Pianzola et al.). 
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Constructing Domains in Visual Narratives: 
Structural Patterns of Incongruity Resolution 


BIEN KLOMBERG, JOOST SCHILPEROORD, NEIL COHN 


Abstract: Understanding visual narrative sequences, like those in comics, requires readers to track 
situational information like the continuity of characters, locations, or events across panels. Yet, some 
sequences intentionally present incongruities, which may elicit the inference that the narrative 
presents two separate domains: an established, expected storyworld (primary domain) and an addi- 
tional context surrounding the incongruous events (auxiliary domain), like a character’s imagina- 
tion, dreams, or memories. This paper describes how these inferences are supported by visual con- 
structional patterns, which emerge across a wide range of narratives, yielding further insight into the 
fundamental role of graphic and structural cues within visual storytelling. 


Keywords: incongruity resolution, domains, parallel architecture, visual language, comics. 


Te understand visual storytelling, readers need to track situational information like characters, 
places, and events across panels, and integrate these coherently in a mental representation of the 
narrative (Cohn, “Your Brain on Comics” 355; Dijk and Kintsch 5; Huffet al. 943; Loschky etal. 
312). Generally, information that is discontinuous with the current context is a cue for an event 
boundary (Cohn, “Your Brain on Comics” 360; Gernsbacher, “Coherence Cues” 3; Loschky et al. 
322), but some sequences may include intentionally incongruous information that needs to be mean- 
ingfully incorporated in the current storyline. How do we deal with such issues of incongruity? 

Consider Figure 1, which shows short comics from JA, a series about the experiences of its two 
collaborative creators (who appear as characters in the final scenes of each strip). In all the strips in 
Figure 1, the final panel is incongruous with the prior series of events, juxtaposing distinct places and 
agents within a layout. This divergent information typically prompts readers to engage in “incon- 
gruity resolution”, i.e., finding an interpretation that explains the incoherence (Schilperoord 2-4, 
18-19; Forabosco 47). For Figure 1, readers can resolve the differences across characters/locations 
when inferring that the first series of events are not actually occurring or present at that moment/ 
place, but function as metaphoric depictions of the “real” characters’ actions and/or feelings. In 
Figure 1a, the mole’s actions are analogous to the woman’s: she bumps into things while “as blind as 
a mole”, seeking the glasses worn by the dog. Similarly, Figure 1b’s piranha (devouring meat quickly) 
and snail (eating a leaf slowly) seem to reflect the eating habits of the man and woman, while Figure 
1c compares the woman catching a spider for the man to a superhero defeating a monster to save a 
civilian. Here, the animals, superhero, and monster would not be physically present in the focal 
characters’ storyworld but can be inferred as additional contexts applied to those characters. The 
recognition of these incongruities and their resolutions are essential to the humor intended by these 
comics (Forabosco 47), and give a reason for why this incoherence was shown (Schilperoord 4): they 
reflect the characters’ inner thoughts and feelings. 
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Figure 1. Examples of incongruous events that can be meaningfully incorporated. 
All sequences are from the series JA! © Angela Cuéllar and Jonds Aguilar (Creative Commons License). 


Intentional incongruities like in Figure 1 have been widely recognized in visual narratives, united 
by the inference that some events do not actually occur at that place/moment. This gives rise to two 
contexts that should be integrated during incongruity resolution: a primary domain (the established 
storyworld, e.g. the man and woman) and an auxiliary domain (an additional context associated with 
events inferred to not occur, e.g. the analogy with animal habits/superheroes). Such inferencing may 
make sequences more complex but also engaging, rewarding, and fun, since readers are challenged 
but can still manage to construe meaning (Ramachandran and Hirstein 31; Giora et al. 138). In 
Figure 1, these domains (and their resolution) provide the strips with their aesthetic appeal. Such 
inferential challenges may also heighten narrative tension, e.g. for horror and mystery genres that 
may deliberately prolong cues that help to resolve incongruity. Since the resolution of these se- 
quences relies on successfully integrating two domains, and such phenomena appear to rely on 
patterned structures, we refer to these as domain constructions, but similar phenomena have been 
subsumed by work on perspective taking or focalization (Maier and Bimpikou; Mikkonen 110; 
Horstkotte and Pedri 38-40). 

Although prior approaches have attempted to describe the semantics related to resolving the 
incongruities within domain constructions (Abusch and Rooth 8; Bimpikou 246; Maier and 
Bimpikou), they have not addressed what visual forms might cue domain constructions, and how 
these cues correspond to meaning or the structural patterns that arise across such features of form. In 
Figure 1, the layout of the three sequences is such a form-related cue: all the panels showing the 
animals’/superhero’s actions create a grouping, while the panel showing the human characters is its 
own row. These groupings are reinforced by panel borders, with the first four/six panels using 
explicit borders, while the final panel has none. Additionally, aspects of form may create connections 
between primary and auxiliary figures. In Figure 1a-b, the eyes of the mole/snail are drawn the same 
way as the woman’s eyes, and in Figure 1b and 1c, the red of the meat/ civilian’s shirt corresponds to 
the man’s shirt, while the green of the leaf/superhero’s top corresponds to the woman’s shirt, and the 
superhero mask in 1c to her red glasses. Overall, formal features align with the semantic incongruity 
and may cue intended relations between domains and their respective figures. 

This paper aims to account for such patterns and how they may signal incongruity to be resolved. 
Our examples include a variety of comics by different authors, highlighting how domain construc- 
tions appear widespread in storytelling for authors to convey characters’ personal experiences, while 
aligning these meaning relations with conventionalized graphic features. The analyses are embed- 
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ded in a cognitive and linguistic approach to visual representations, and as such, aim to describe the 
patterned nature of these encoded constructions. We thus investigate the effects of graphic and 
structural cues on (inferential) meaning-making in storytelling and what constructional patterns 
persist across authors and their works. Additionally, such patterns could apply to other visual media, 
e.g., film, which maintain many of the same affordances as drawn visual narratives. Naturally, 
creators may also rely on similar schemas when working across media, often because film storytelling 
first occurs as drawn storyboards. To discuss domain constructions for comics, we first present how 
visual narratives are processed and how previous work has described issues of domain constructions 
similar to Figure 1. Next, we propose a model that describes patterned correspondences between 
form and meaning related to this construction. 


Domain constructions in visual narratives 


To comprehend visual sequences in general, readers integrate story components into one coher- 
ent, mental representation of the narrative (Cohn, “Your Brain on Comics” 355; Dijk and Kintsch 
5; Huffet al. 943; Loschky et al. 312). This representation is known asa situation model, and includes 
situational aspects such as characters, their motivations, spatial locations, time, etc. (Dijk and Kintsch 
46; Zwaan and Radvansky 167; Gernsbacher, “Coherence Cues” 5). With each incoming piece of 
information, the situation model gets updated, and sufficiently congruent elements can be mapped 
onto the existing model (Cohn, “Your Brain on Comics” 360; Gernsbacher, “Coherence Cues” 3; 
Loschky et al. 321). The more incongruent the incoming information, the greater the updating costs 
(Huffet al.; Hutson et al.; Magliano et al.), until significant discontinuity sponsors an event boundary 
(Gernsbacher, “Two Decades” 5; Loschky et al. 322). 

While sequences with auxilairy domains are incongruous and have been shown to require more 
updating indeed, these were equally comprehensible as sequences with only a primary storyworld 
(Klomberg, Fadeeva, et al.). Moreover, domain constructions seem to be fairly common, with cor- 
pus work showing they are prevalent in around 65% of comics (van der Gouw et al. 24). The 
ubiquity and comprehensiblity of domain constructions give rise to the question of what mecha- 
nisms underlie this storytelling technique. 

Prior work converges on a general semantic distinction between visual events that are perceptible 
to all characters versus those that are perceptible only to a single character (Abusch and Rooth 8; 
Bimpikou 247; Duncan 275; Horstkotte and Pedri 13, 147-48; Maier and Bimpikou). To mark this 
distinction, we use the notion of domains (Clausner and Croft 2; Langacker 147), which denote 
particular types of mental spaces (Fauconnier and Turner 40) within the context of narrative se- 
quencing, and thus a certain context that visual events can be attributed to. The primary domain 
represents events that are inferred to transpire in the “actual” storyworld, i.e., the world physically 
accessible or perceivable to all characters. An auxiliary domain on the other hand, represents events 
inferred as “not physically occurring at that time”. Auxiliary domains are typically resolved as 
depictions of a (private) mental experience of certain character, such as their fantasy, hallucina- 
tions, dreams, or memories (Horstkotte and Pedri 40-48; Abusch and Rooth 12; Maier and Bimpikou; 
Bimpikou 255; Mikkonen 110). Work on focalization (Horstkotte and Pedri 1) describes domain 
constructions by analyzing how a narrative may present the experiences of a certain character, while 
approaches from semantics and discourse theory focus on the veridicity of events (Bimpikou 247; 
Maier and Bimpikou). 

In addition to semantic distinctions between events, prior work foregrounds the role of perspec- 
tive-taking for these sequences (Abusch and Rooth 8; Bimpikou 249; Maier and Bimpikou), with 
inferences of subjectivity as cue for auxiliary domain events. Consider the modified example in 
Figure 2a, exemplifying a two-panel construction that implies subjectivity. When a panel with a 
character looking off-panel (the man in red) is followed by an object incongruous with the narrative 
(the snail), the sense of perspective-taking evoked by this two-panel construction facilitates the 
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inference that the incongruity is a perception private to the character, i.e. their imagination, dream, 
or hallucination (Abusch and Rooth 8; Bimpikou 248-49; Maier and Bimpikou). This contrasts 
with “blended-perspective” (Bimpikou 246; Maier and Bimpikou), where elements of distinct do- 
mains are merged ina single panel. In Figure 2b, a man looks into the window of a shop but sees a 
reflection of a different person, with no apparent co-reference. Contextually, only the man can see 
the reflection of this dark-haired figure, who also haunts him in his dreams later on. Thus, this 
reflection seems to be imagined by the man, as if he and this figure were one in this moment 
(reinforced by the comic’s title: Bonding). In this way, blended-perspectives merge the physical 
storyworld (the shopping street) with the private thoughts of a character (the reflection of the dark- 
haired figure). 


Figure 2. Examples illustrating a) a two-panel construction and b) a blended perspective. Figure ais a 
modified version of a sequence from the series JA! © Angela Cuéllar and Jonas Aguilar 
and Figure b from Bonding © 2015 Lumarin and Marina Privalova. 


While the availability ofan identifiable experiencer may be helpful for the recognition of domain 
constructions, not all auxiliary domains can be readily interpreted as a particular character’s experi- 
ence. For Figure 1a-c, it is unclear whether these sequences are imagined by a storyworld character 
(does the woman imagine herself'as a mole/snail/hero?) or a comparison made by a non-diegetic 
narrator. Despite this conceptual ambiguity of why the two event types are integrated together, the 
cues sponsoring those inferences (e.g. layout, visual patterns, colour correspondences, etc.) can be 
identified without issues. It thus seems that these cues can function independently from meaning. 

These and other patterns have received little focus across research on visual narratives, despite 
some prior observations of cues for individual images. For instance, previous analyses acknowledge 
that divergent panel borders or changes in layout correspond to incongruous events (Duncan 275; 
Horstkotte and Pedri 40-46; Mikkonen 110-11), but they have not been generalized beyond the 
specific examples that are addressed. Moreover, there are patterns not yet accounted for, e.g. Figure 
1’s segmentation of panels and its ambiguous (or lacking) experiencer. Thus, while existing theories 
have yielded valuable insights into the meaning-making processes involved in domain resolution, 
these do not acccount for the formal patterns that signal these construals, which may give the false 
impression that such construals appear without patterned contexts. How then can we account for 
these various visual narrative constructions? 
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The Domain Constructions model 


To account for the structured patterns of form and meaning that occur in domain constructions, 
we here propose the Domain Constructions model, or DC model. The model describes structural 
correspondences between form and meaning that occur for domain constructions, where incongru- 
ity across situational aspects and form-related features cue the search for a meaningful interpreta- 
tion. This involves integrating an auxiliary domain (events inferred to not physically occur at that 
time/place) within the story’s primary domain (the physical storyworld accessible to all characters). 
The proposed model is not restricted to cues signalling domains across panels but also pertains to 
elements within panels (Bimpikou 245-46; Horstkotte and Pedri 40-42; Maier and Bimpikou). 

The DC model is embedded within the broader framework of Visual Language Theory (Cohn, 
The Visual Language 1-13) which posits that graphic communication maintains structures and cog- 
nition consistent with natural languages, as has been supported both theoretically and empirically (see 
Cohn, Who Understands Comics? (20-22)). Visual Language Theory itself is embedded within the 
linguistic model of the Parallel Architecture (Jackendoff 107-26), which characterizes language in 
terms of three parallel, but independent structural components: modality (phonology), grammar (syn- 
tax), and meaning (conceptual structure), as well as interfaces that emerge between these. The Paral- 
lel Architecture has been applied to characterize visual language in terms of the same components of 
modality (graphic structure), grammar (combinatorial structure), and meaning (conceptual struc- 
ture) (Cohn, Who Understands Comics? 5-6). These three structures, shown in Figure 3 below, 
describe both single units, e.g. a single panel, and sequences of units, e.g. multiple panels on a page. 


Modality: Grammar: Meaning: 
Graphic structure Combinatorial structure Conceptual structure 


Graphological Morphological Conceptual 
structure TT structure structure 
Unit-level Unit-level 
Layout —_— > Narrative . Event 
grammar structure 
Sequence-level Sequence-level 


Figure 3. The Parallel Architecture framework. 


In visual expressions, each component involves unit- and sequence-level structures, indicated by 
the vertical arrows in Figure 3. The modality component (i.e., graphic structure) related to single 
panels is termed graphological structure (Klomberg, Hacimusaoéglu, et al. 10), which characterizes 
the organization of the visual markings that make up the images, such as lines, dots, or colours, and 
the visual regions they compose. Graphic structure across panels is captured by layout, which ac- 
counts for how sequences of images are integrated spatially on a canvas. The grammar component 
characterizes the combinatorial structure of a single panel in terms of its morphological structure, 
and organizes sequences of panels according to principles of Narrative Grammar (Cohn, Who 
Understands Comics? 12-16). Finally, the meaning component describes how panels are understood 
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independently (conceptual structure) and in sequence (event structure). For domain constructions 
such comprehension relies on similar principles, as the same conceptual inferences apply within and 
across single panels. 

While domain constructions would be resolved at the level of conceptual structure, the DC model 
accounts for the ways that graphic and combinatorial structures contribute significantly to their 
recognition and resolution; these interfaces across components are expressed by the horizontal arrrows 
in Figure 3. Hence, the following sections will detail how constructions encoded across these com- 
ponents signal auxiliary domain usage and/or support relations of co-reference across entities that 
initially appear non-co-referential. We first present structural correspondences between the graphic 
and conceptual structures, then between grammatical and conceptual structures, and finally, struc- 
tural correspondences across all three components. 


Graphic Structure 


Graphological Structure. To describe how form may interact with meaning we must first 
characterize the graphic form of visual representations. Analogous to how phonological structures 
organize sound structures, a graphological structure (Klomberg, Hacimusaoglu, et al. 10) organizes 
visual-graphic components. The basic units of those components are graphemes, i.e. dots, lines, 
curves, and spirals, which combine to create larger structures of “regions”: enclosed visual groupings 
(either explicitly so or inferred) (11). Regions, in turn, can recursively combine to form even larger 
graphic structures that correspond with conceptual representations, i.e., they can be recognized as 
meaningful objects (Willats 94). Graphological structure is thus concerned with purely graphic 
information, such as the lines and proporties of regions. We here propose that differences across 
these graphic features are often used to signal domain constructions. 


Figure 4. Examples illustrating graphological cues corresponding to meaning. 
Figure a is from Socrate © Francesco Barilli and Alessandro Ranghiasci, 
Figure b from Jar of Fools © Jason Lutes, and Figure c from Raruurien © Ann Maulina. 
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Figure 4 illustrates three types of graphicological cues that may signal different conceptual do- 
mains: colour, linework, and shape similarity (Horstkotte and Pedri 12; Mikkonen 116-20). Consider 
Figure 4a, where the first page shows the philosopher Socrates during his own trial, the second page 
presents Socrates amongst different people in a different location, and the third page shows different 
figures in an unknown environment. These differences in both characters and locations confront the 
comprehender with an incongruity. Crucially, each page here is marked with a distinct colour scheme. 
The incongruity across pages can be resolved by construing the second page’s events as taking place at 
another moment (Socrates teaching his students), with the same agent in both scenes (Socrates). The 
third page can subsequently be construed as the contents of Socrates’ lesson. These inferences are 
signalled by the pages’ graphological differences, with the first page’s trial (the primary domain) 
shown in colour, the second page’s memories (an auxiliary domain) in black and white, and the third 
page’s exposition (a second auxiliary domain) in black and orange hues, like a Greek vase (Andrew 
et al, 2). A similar alignment of colour schemes and domains occurs in Figure 4c, where the final 
incongrous panel can also be inferred as a past event featuring the same agent (the woman). This 
panel is coloured only in blue hues, as opposed to the range of colours used in previous panels. 

Differences in the linework of drawing styles may also correspond to domain constructions. In 
Figure 4a, the third page’s drawing style further mimics Greek vase art , with figures shown mostly 
from the side and without shading (Andrew et al. 5). This stylistic shift again aligns with the concep- 
tual interpretations that arise for page 2 and 3. While both pages show colour differences that 
correspond to different domains, page 2 maintains the same drawing style as the one used for 
primary domain events on page 1, warranting the interpretation that the memory event bears a 
substantive conceptual relation to Socrate himself, i.e. his lived experience. In contrast, page 3’s 
features elicit associations to the myths and legends typically shown on those Greek vases (Andrew et 
al. 12), which aligns with the idea that this event is fictitious even to Socrates himself. Seemingly, the 
use of differing drawing styles across pages 2 and 3 in Figure 4a functions to distinguish between the 
two different auxiliary domains. 

Graphological differences signalling different domains also emerge from the linework of the panel 
borders. In Figures 4a-c, diffferent panel borders may support inferring those panels as correspond- 
ing to auxiliary domains. Figure 4a’s third page has thick, decorated borders which contrast the 
previous two pages’ borders, thus emphasizing that this last page may present a second auxiliary 
domain rather than a continuation of the memory on page 2. In Figure 4b, the use of different 
borders likewise supports a distinction between the events captured in those panels: the man dreams 
that he is dancing with the woman (shown in panels with rounded borders), and wakes up to find he 
is actually sitting in a park (shown in panels with square borders). In Figure 4c the final panel has 
distinctly frayed borders, compared to the previous rectangular borders, reinforcing that the final 
event is part of an auxiliary domain (here, a memory). Besides these differences in panel borders’ 
linework, differences can also arise when borders are omitted, as in Figure 1. Overall, incongruity in 
lines across panels may signal that additional updating is needed. 

Finally, similarities in shapes across panels can motivate different domains. In Figure 4b, the switch 
between domains occurs between panels 8 and 9, which both depict the main character. Noticeably, 
the character’s composition is highly similar across panels, with his hands in a similar position 
relative to his face despite changes in his expression, hair, clothing, and background. The overall 
similarity alongside small differences seems to emphasize that this same moment occurred across two 
domains (him holding up his hands in a dream and in the physical storyworld). Experimental re- 
search (Klomberg, Fadeeva, et al.) suggests that familarity with comics may mediate whether such 
contour similarity affects processing, with more experienced readers slowing down for sequences 
lacking this similarity. 

As shown by our analyses, these graphological cues do not indicate the presence of auxiliary/ 
primary domains themselves, but rather do so in contrast to other panels (Mikkonen 119). For 
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example, a whole sequence can have rounded, wavy, or no borders or be shown in greyscale or blue 
hues, but this alone does not signify auxiliary events. When these features contrast other panels that 
have been established as normative to a primary domain, it may prompt readers to infer the events 
with incongruous features as part of an auxiliary domain. These graphological patterns thus include 
open slots, filled with whatever applies to a specific comic (e.g. Figure 4b has rounded borders as 
incongruent vs. square borders as expected, while Figure 4c contrasts wavy vs. square borders). Such 
cues often apply to several panels; e.g., Figure 4a-b’s colours schemes and divergent borders persist 
for all panels depicting those auxiliary domain events. Additionally, graphological features can easily 
combine, whether border differences and shape similarity (Figure 4b), colour and border differ- 
ences (Figure 4c), or colour, style, and border differences (Figure 4a). 

Layout. Layout refers to the part of the graphic structure that concerns the spatial organization of 
panels on a larger canvas (e.g. a page) (Cohn, Who Understands Comics? 9-11). We already de- 
scribed one pattern of layout for Figure 1, where the auxiliary domain panels cluster together 
followed by a single panel showing the primary domain. This general construction is shown in 
Figure 5a where tree structures are used to specify the organization of these sequences. 


Canvas 


Vertical/Horizontal Panel* (Vertical/ Horizontal) 


Canvas Canvas 


Vertical 5° Horizontal 3* Horizontal 


Horizontal Horizontal 


Figure 5. Examples illustrating layout trees with the * indicating the introduction of asecond domain 
(whether primary (b) or auxiliary (c)). Figure b is slightly adapted from the series JA! © Angela Cuéllar and 
Jonas Aguilar, and Figure c is slightly adapted from REAL © Inoue Takehiko. 
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Figure 5b shows how Figure 1’s examples would fit this organization: the events in one domain 
(panels 1 to 4) are structured as multiple horizontal rows stacked on one another vertically. This 
column of panels then combines with another row underneath that corresponds to the second do- 
main (panel 5), forming an ultimately vertical layout. This construction can have additional internal 
structure within its lower-level vertical grouping; e.g., Figures 1a and 1c feature two rows, while 
Figure 1b features three. Note that Figure 5b is ambiguous: it could also be viewed as three horizon- 
tal rows stacked upon each other, rather than a column upon a row. Nevertheless, either interpreta- 
tion maintains the general construction illustrated in Figure 5a and demonstrates a recurrent pattern 
of segmentation across panels that aligns with conceptual inferences, with a sequence’s second do- 
main corresponding to a grouping on its own. 

A second instantiation of the general pattern is shown in Figure 5c, where the horizontal row 
combines with other horizontal structures not only before but also after. Here, a boy ina hallway 
(panels 1-2) remembers himself and a girl on a motorbike (panel 3). This panel on its own (panel 3) 
corresponds to an auxiliary domain of a past event. In Figure 4a, different domains are distributed on 
different pages. Thus, it seems that layout segmentation may support conceptual interpretations of 
domains by presenting distinct panel groupings within page structures as well as across pages. Such 
segmentation echoes other observed alignments between coherence shifts and layout structure 
(Hacimusaoéglu et al.). 

All the patterns discussed so far strictly concern the organization of panels and/or pages, and render 
available slots that can be filled by particular forms, rather than specific meaning. In Figure 5b, the 
single panel functioning as a row corresponds conceptually to a primary domain, while in 5c, the 
single panel corresponds to an inference of an auxiliary domain. This emphasizes that these con- 
structions are aspects of layout which can interface with meaning in various ways. These patterns 
describe how the narrative switches domains in general, regardless of the direction of this switch, or 
the particular nature of the domains involved (Klomberg, Fadeeva, et al.). 

Nevertheless, one layout structure appears to relate to auxiliary events more directly. Figure 6 
shows two collage layouts, where multiple scenes graphically overlap without overt panel borders. 
Here, both pages show events that can be inferred to have taken place in the past and can thus be 
identified as auxiliary domain events. Collage layouts potentially emphasizes the sense that such 
events are “flashes” of memory. Corpus work indeed found that 83.8% of collage framing contained 
auxiliary domains (van der Gouw et al. 36), supporting a close relationship between them. 


Figure 6. Examples illustrating collage sequences. Figures are from Valley of White Birds 
© Wolfsmoke studios and from Pirozhki © Sideburn004, respectively. 
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Combinatorial Structure 


Morphological Structure. This component of the Parallel Architecture is concerned with the 
patterns of abstracted forms of visual units (Cohn, The Visual Language 23-24; Cohn, “Combinato- 
rial Morphology”; Cohn, Who Understands Comics? 12). Similar to morphology in verbal language, 
visual morphology contains units of visual language that can stand on their own (e.g. a single char- 
acter or object) next to bound morphemes that need to combine with units to be meaningful. These 
components use various strategies to combine, such as via suppletion, e.g. replacing a character’s 
eyes with hearts to show love, or blending, e.g. depicting a character’s normal body with an animal’s 
head (Cohn, The Visual Language 44-47; Cohn, “Combinatorial Morphology”). 

These morphological patterns may signal domain switches and semantic (co-referential) connec- 
tions across characters from different domains. Figure 7a illustrates a gradual replacement (partial 
suppletion), where the man’s increasingly disturbing facial features from panel 4 to 7 signal that the 
woman slowly realizes his malicious intentions. Figure 7b’s morphology signals the start of a dream: 
a man was watching a movie with a female protagonist (panels 1-3), but evidently fell asleep as the 
movie becomes personal and nightmarish, and he wakes up on a later page. He apparently falls asleep 
around panel 4, when the actress’ appearance blends with his own. This “morphological blend” 
shows parts of one meaningful entity (the woman’s hair, clothes, and earrings) merged with parts 
from another entity (the man’s own face and body hair), which supports the inference that this is a 
dream rather than a movie. Finally, Figure 7c shows visual components repeated across different 
figures, with the woman and animals having the same eyes and eyebrows. These we term “echoic 
morphemes”: visual components that mimic one another morphologically. Rather than signalling a 
domain switch, repeating visual components may signal readers to infer some semantic relation 
between characters, and as such, may support subsequent inferences of co-reference across these 
units (Klomberg, Hacimusao&lu, et al. 21). 


a) Partial suppletion b) Morphological blend 


Figure 7. Examples illustrating the morphological strategies of a) partial suppletion, b) morphological 
blend, and c) echoic morphemes. Figure a is slightly adapted from Vicki Lante © Patrick Steptoe, Figure b is 
slightly adapted from Jar of Fools © Jason Lutes, and Figure c is slightly adapted from JA! 
© 2016 Angela Ciiellar and Jonas Aguilar. 
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Narrative Structure. Where morphology is concerned with the relations between visual units 
within panels, narrative structure characterizes the combinations of panel units, along with the 
function these panel units play within a narrative sequence. The combinatorial structure of se- 
quences is captured by Visual Narrative Grammar (VNG) (Cohn, “Visual Narrative Structure” 
420-38; Cohn, Who Understands Comics? 43-46), which identifies the narrative categories of panels 
and the hierarchic embedding of larger sequences of panels. We first explain the narrative categories 
panels can have, and then analyze how these categories may interface with domain switches. 

Figure 8 shows the narrative structure of the comics in Figure 1 according to VNG (Cohn, “Visual 
Narrative Structure” 420-27; Cohn, Who Understands Comics? 43-46). Let’s start with the first 
grouping in each of these sequences. In Figure 8a, panel 1 introduces the scene and characters (an 
Establisher), and then panel 2 sets up (an Initial) the climactic moment in panel 3 (the Peak), until this 
tension diminishes in panel 4 (a Release). These four panels follow the canonical narrative schematic 
order (E-I-P-R) within VNG. Figures 8b and 8c use an additional “conjunction schema” where 
identical categories follow one another. Figure 8b uses an action-conjunction (see subscript a), 
where panels of the same narrative categories repeat similar actions. Figure 8c shows an environ- 
mental-conjunction (subscript e), where the depicted characters or objects are inferred to belong to 
the same environment. 

Narrative categories not only apply to individual panels, but also to constituents of panels. In 
Figure 8a, the first four panels constitute the climax of the sequence, thus forming a Peak sequence, 
while the final panel diminishes that tension and acts as a Release for all preceding panels. Figure 8b 
and 8c use this same pattern, with a final Release panel following a Peak segment. 


a ) Arc 


Peak Release 


Establisher _ Initial Peak Release 


b) Arc 

Peak Release 

Initial Peak, Release 
Peak Peak 
7" 
y 
c) Arc 
Peak Release 
Initial, Peak, Release, 


Initial Initial Peak Peak Release Release 


Figure 8. Examples illustrating grammatical tree structures with a a) canonical scheme, b) action conjunc- 
tion, and c) environmental conjunction. Figures are slightly adapted sequences from JA! 
© 2016 Angela Cuellar and Jonas Aguilar. 
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For these sequences, narrative constituents correspond with the conceptual division between 
domains. Note that this division persists despite internal variation within the constituents, with the 
Peak-Release sequence on the highest level corresponding to a division between auxiliary and 
primary domain events. Other narrative constructions, e.g. conjunctions, could similarly support 
domain constructions across panels, and will be investigated more extensively in future works. As 
shown for other structures, these patterns can be expected to occur regardless of what conceptual 
domain fills what slot. 


Meaning 


Conceptual Structure. Conceptual structure specifies how representations are involved in mean- 
ing-making, which in the case of domain constructions often involves complex inferencing. Incon- 
gruity occurs across situational information, with time, space, or characters mismatching across panels, 
signalled also through differences in graphology, layout, morphology, and grammar. Such incongru- 
ity can then prompt the resolution that those depictions reflect events that do not actually occur in the 
storyworld (the primary domain), but instead correspond to some alternative context (an auxiliary 
domain). All examples shown so far involved such inferencing. Despite deviating from the primary 
domain, resolving these incongruities allows for coherent integration into situation models. 

The way that domains are to be integrated to arrive at a sensible resolution will depend on the 
inferrred nature of the auxiliary domain, i.e., imagination, hallucinations, dreams, and memories 
(Abusch and Rooth 12; Bimpikou 255; Horstkotte and Pedri 40-48; Mikkonen 110-12). For instance, 
incongruent appearances ofa character can be resolved as happening ina dream in which the character’s 
appearance transforms, rather than this character actually (physically) changing or that a new charac- 
ter is being introduced. Particular graphic and/or structural patterns may be used for certain types of 
auxiliary domains; e.g., greyscale panels seem more indicative of memories than hallucinations, for 
which we may instead expect bright or atypical colours. So far, the DC model categorizes nine types 
of auxiliary domains: Imagination, Knowledge, Dream, Memory, Foresight, Hypothetical, Fiction, 
Fourth Wall Break, and Analogy, along with cases that may remain Ambiguous. 

Imagination and Knowledge domains are inferred as events that occur in the mind of a single 
experiencer during an active (awake) state. The page in Figure 9a (read from right to left) precedes 
the “blended-perspective” from Figure 2b, and implies the reflection is imagined: the man with the 
cap (the experiencer) sees an incongruent reflection in the shop window (panel 3), which only he 
appears (able) to see, as the reflection disappears when someone else is present (panel 6). Generally, 
Imagination domains show fantasized events originating from a single, awake character that other 
characters seem unable to perceive, e.g. active imagination, daydreams, or hallucinations. In Figure 
9b, the swimming techniques can also be inferred to occur in the mind of the boy only, due to the 
juxtaposition of these techniques and the close up of the boy’s face, and reinforced by the absence of 
physical materials he could be looking at in the rest of the sequence. Rather than a fantasized event, 
however, these seem a visual representation of the boy’s knowledge relating to specific techniques. 
While both domains appear private to one experiencer, Imagination typically depicts made up 
information, while Knowledge reflects thoughts ofa more substantive nature, such as spatial knowl- 
edge (e.g. a map of the environment), encyclopedic knowledge (e.g. how a certain object looks or 
functions), or instructions (e.g. how to perform certain actions). 

While these events occur during characters’ awake state, experiencers may also visualize events 
involuntarily, e.g. when dreaming. In Figure 9c, the third panel shows the experiencer jolting up- 
right with a scream, suggesting he awakens, which is emphasized by the sixth panel depicting him on 
his bed. The incongruity between the first two panels and the rest can thus be resolved by inferrring 
the first two panels to represent a dream. Typically, Dream domains are preceded by a character 
lying down or closing their eyes and/or followed by a character wakening or opening their eyes, 
providing contextual cues that signal the incongruous events as part of a dream. 
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Figure 9. Examples showing an a) Imagination domain, b) Knowledge domain, and c) Dream domain. 
Figure a is from Bonding © 2015 Lumarin and Marina Privalova, Figure b from 
Poisson © Nie Jun, and Figure c from Jar of Fools © Jason Lutes. 


These three domain types concern events assumed to originate completely from someone’s mind, 
but two other types -Memory and Foresight- denote events that should be inferred to occur at some 
past or future point in the story’s timeline. Figure 4a (Socrates’s trial), Figure 4c (the woman remem- 
bering being angry), and Figure 6 (the collage examples) all show examples where incongruous 
events have to be construed as a protagonist’s memories. Similarly, the collage panel in Figure 10a 
shows the protagonist overlayed with a series of events that have occurred before, some of which 
even have been shown in the story explicitly. In these instances of Memory domains, the primary 
assumption is that the depicted moment is something that the experiencer recalls from the past 
(which may or may not have been shown earlier in the story). Narratives can also show events that 
will occur in the future, e.g. via a prophecy or a character having a vision. Figure 10b shows an 
example of a witch watching her crystal ball with the vision illustrated on the right (a princess 
picking up a sword, coaxed by the snake on her shoulders). This exemplifies a Foresight auxiliary 
domain, representing (likely) future events. 
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Figure 10. Examples showing aa) Memory domain, b) Foresight domain, and c) Hypothetical domain. 
Figure a and b from Valley of White Birds © Wolfsmoke studios and Figure c © Sharitha van der Gouw. 


In addition to past and future events, a depicted event may or may not come to pass, i.e., a 
Hypothetical auxiliary domain. Examples are thought experiments (an answer to the question “What 
if...2”) or outcomes related to very specific circumstances. In Figure 9c the first panel shows an old 
man saying that “the white goddess” (i.e. the white creature in the second image) may be caught off 
guard only by the protagonist in panel 2 and 3, and so he “might just be the only one who could kill 
her”. The panel in between these speech balloons shows such a scene where the protagonist has killed 
the white goddess. However, the reader knows this event has not (yet) happened, and so the panel 
shows an Hypothetical situation (cued predominantly by the modal keyword “might”), without 
indicating whether the event is likely to occur or not. 

Thus far, we discussed auxiliary domains representing events revolving around the experiencer 
that visualizes, remembers, or otherwise inspires that event. However, auxiliary domains can also 
represent events that do not appear private to an experiencer. The first three panels of Figure 11a 
show different moments of a fight, while the final panel shows a person holding a comic book 
showing these exact three panels. Here, the first three panels depict fictitous events (a comic book 
narrative), and the final panel depicts the primary storyworld domain, where the story will continue. 
This auxiliary sequence is thus part of a Fiction domain, which can be described as a ‘story within a 
story’ and/or events that are fictitious to the characters themselves. Examples include depictions of 
folklore history, fairytales, or other accounts of (past) events that do not involve the narrator itself but 
other (fictitious) people. 
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Figure 11. Examples showing a a) Fiction domain and b) Fourth Wall Break domain. 
Figure a is from El Verdagazo © Jorge E. Pérez and Francisco Alberto Lopez 
and Figure b from Cottonstar © Ben Geldenhuys and Danelle Malan. 


Additionally, narratives may also suspend the established storyworld altogether, as in Figure 11b 
where the first panel shows the comic creators reacting to a story event on the previous page, while 
the second panel continues in the primary storyworld domain. Thus, this first panel steps out of the 
story fora moment and visualizes figures with no relation to the narrative at all (except supposedly 
having created it). The incongruity across figures and places is resolved when the first panel is 
interpreted as an event from the creator’s own world. We summarize these as instances of a Fourth 
Wall Break domain. It is worth pointing out that this also allows visual narratives to be self-referen- 
tial, and to have creators comment on their own creation. 

The last type of auxiliary domain is Analogy, which resolves incongruity via a (metaphoric) 
comparison between two domains. This was the case for Figure 1, where the animals and superhero 
figures metaphorically represented the actions and/or feelings of the actual characters shown in the 
final panel. For these domain constructions, it may be ambiguous whether an diegetic experiencer is 
behind the analogy (i.e., whether the woman imagines herself as a mole/snail/hero) or whether the 
comparison can be attributed to a non-diegetic narrator/author. 

Finally, auxiliary domain events cannot always be easily categorized and may evoke multiple 
possible interpretations. A sequence may not imply an experiencer explicitly, e.g. Figure 1, or Figure 
10c, where the Hypothetical situation of the protagonist killing the white goddess could be attrib- 
uted to the mind’s eye of either the protagonist or his mentor (or both). Even sequences with overt 
experiencers may be Ambiguous, when it is unclear what action the experiencer is involved in 
(imagining, remembering, etc.). In those cases, contextual cues then do not distinguish whether the 
depicted scene is private to the experiencer or not, or whether the depicted scene could have already 
occurred in the past. Alternatively, cues may overlap, such as when a characters’s dream features a 
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memory or likely future event. This ambiguity is likely presented on purpose to leave readers in 
suspense or create more possible inferences (e.g. is ita dream or a prophetic vision?), requiring (or 
granting) more reader involvement and freedom to interpret sequences according to readers’ own, 
strongest associations, which may contribute to the pleasure of reading. 


Meaning across modalities 


The discussion so far related to visual information; however, discontinuities also persist in 
multimodal interactions between text and images. Various examples illustrate such an interaction. In 
Figures 4c, 6a, and 9d, the text includes past tense and deictic indicators (4c: “that time”, 6a: “back 
then”, 10a: “my life like this. ..has been”), supporting the interpretation that the visual image depicts 
a past event. Figure 10c is even more overt, with the modal verb “might” reinforcing the inference 
ofa Hypothetical domain. In Figure 11b too, the text in the first panel identifies who is depicted (“the 
creators of this comic”), which would have been difficult to establish otherwise. These references 
may even work in hindsight or advance, e.g. when auxiliary events are referenced in later panels 
(e.g., “I had a bad dream”), textully establishing an event as a particular domain type (here, Dream). 


Interfaces across Parallel Architecture structures 


So far, we have discussed how the DC model operates on correspondences between graphic 
structure and meaning, and between combinatorial structure and meaning, and how both support 
understanding of domain constructions in visual narratives. Correspondences across these structures 
may align in various ways for visual narratives, which gives rise to different ways that auxiliary 
domains are presented within a sequence. 


Figure 12. Illustration of the three domain switches: a Sudden transition, 
a Gradual transition, and a Partial transition. 
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Figure 12 summarizes how narratives may switch between their multiple domains. With a Sud- 
den transition, layout constructions align with grammatical constructions, such that the narrative 
switches domains from one individual panel to another. In the example below, the narrative switches 
to a memory event across panels 3 and 4, with each domain reflected in its own panel. This type of 
transition would also apply to the two-panel constructions discussed by Abusch and Rooth (8). For 
Gradual transitions, the domain switch requires more than two panels. The example below illustrates 
how change across panels occurs progressively, with each subsequent panel amplifying the previous 
one until a certain end stage is reached (here, the woman realizing the man’s malevolence). Finally, 
a Partial transition relies on graphological and morphological patterns rather than layout and gram- 
mar, since two domains are merged within a single panel rather than across a sequence, what previ- 
ous studies term “blended-perspective” (Bimpikou 246; Maier and Bimpikou). Here, auxiliary 
elements are added to the primary domain, such that we assume the narrative remains in the estab- 
lished storyworld overall but overlays this scene with auxiliary aspects (the example here shows an 
Imagined reflection within an otherwise primary domain environment). 


Discussion 


Altogether, the Domain Constructions (DC) model describes how graphic, combinatorial, and 
conceptual structures contribute to construing incongruity in visual narrative sequences as depic- 
tions of auxiliary domains, such as character’s (private) thoughts or other mental processes (e.g. 
dreams). Inferences of such specific predicates thus signal that the incongruity was likely intentional, 
meant to distinguish these events from (congruous) events belonging to the established, expected 
storyworld accessible to all characters, i.e. the primary domain. 

The DC model expands upon prior (semantic) analyses of intentional incongruity by arguing that 
graphological, layout, morphological, and grammatical structures can all cue the conceptual inter- 
pretation of an auxiliary domain. For graphic structure, these include correspondences between 
meaning and colour schemes, (panel border) linework, or layout; within combinatorial structure, 
morphological strategies distinguish domain switches or connect seemingly distinct figures, and 
narrative patterns may align with domains. Interfaces across the three components then differentiate 
how domains are presented in a sequence, characterized in three types of domain transitions. In 
addition, these constructions appear together in visual sequences, inviting hypotheses related to the 
(mis)alionment of varying cues, e.g. that the alignment of multiple cues could facilitate the inferencing 
related to auxiliary domains. 

As demonstrated in the range of examples throughout this paper, correspondences between form- 
related cues and meaning appear to be encoded patterns across visual narratives, emphasizing how 
incongruity forms a fundamental part of visual storytelling, in addition to congruity. Many visual 
naratives are analyzed for their contingency across discourse units (Maier and Bimpikou; Abusch), 
with discontinuity signalling event segmentation (Gernsbacher, “Two Decades” 5; Loschky et al. 
322). However, the prevalence of incongruous information-semantic, graphic, and combinatorial— 
supports that such constructions are also central within visual storytelling. In addition, this also 
challenges the notion of a narrative as a temporal succession of events (Genette 25), as some depic- 
tions of experiences in domain constructions do not align with the established timeline (e.g. depic- 
tions of past/future events). While these can be construed as e.g. memories recalled in the current 
timeline, such inferences follow resolution based on (temporally) incongruous depictions. The ques- 
tion follows what effect such incongruity has on readers’ narrative experience. As these sequences 
can still be comprehended well (Klomberg, Fadeeva, et al.) we speculate this complexity contributes 
to a sense of intruiging, artful storytelling. 

Furthermore, if indeed visual narrative comprehension relies on such patterned structures, in- 
volving slots to be filled with appropriate content, it suggests authors and readers have learned these 
patterns, and that these might be modulated by visual fluency, as shown for other aspects of visual 
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languages (Cohn et al. 28; Cohn and Kutas 274; Cohn and Maher 79). Such learned constructions 
may have changed over time or differ across genres or cultures. Some constructions may even 
extend beyond comic narratives, depending on the affordances that these require. Colour differ- 
ences, for instance, easily apply to film, while characteristics of panel borders remain a unique 
eraphologial feature of drawn images. Overall, these questions provide relevant challenges for fu- 
ture corpus and/or experimental studies into the comprehension of domain constructions, within the 
studies of comics, discourse, focalization, perspective taking, and cognitive linguistics. 

All in all, the DC model provides a framework for constructions of domains in visual narratives, 
where graphic, combinatorial, and meaning structures combine as formal parallel components under- 
lying a holistic interpretation of incongruity within sequences. These structures capture the com- 
plexity of visual narrative storytelling and its patterned nature, and may easily connect to how 
domain constructions could operate across modalities. The implications of such patterns raise rel- 
evant questions for future work into visual narrative comprehension. 
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Fiction, Empathy, and the Material World 


ROSE TURNER 


Abstract: Fiction may function to support human social interaction by cultivating empathic abili- 
ties. The past decade has yielded promising evidence in support of this theory, though the multidi- 
mensionality of both fiction-engagement and empathy have presented methodological challenges 
and led to mixed findings. Studies have tended to focus on reading and have generally treated 
cognition as a solely internal process. I position empathy and engagement with fiction as ontologically 
extended processes. I argue that further systematic exploration of fiction technologies would pro- 
mote a comprehensive and culturally relevant account of the relationship between fiction-engage- 
ment and empathy and enhance understanding of its cognitive architecture. 


Keywords: Empathy, fiction, social cognition, extended mind, material engagement theory 


Fiction, Empathy, and the Material World 


What a miracle it is that out of these small, flat, rigid squares of paper unfolds world after world after 
world, worlds that sing to you, comfort and quiet or excite you. Books help us understand who we are 
and how we are to behave. They show us what community and friendship mean; they show us how 
to live and die. (Lamott, 1994). 


iction is big business (Nettle, 2005) and a ubiquitous pastime (Loughborough University, 2015; 

Office for National Statistics, 2012; see also, Barnes, 2012). It has been hypothesised that fiction, 
which appears to be species-typical and universal, is a nonutilitarian by-product of other evolved 
capabilities (Pinker, 1997), but it may in fact have an adaptive function (Nettle, 2005; Tooby and 
Cosmides, 2001), benefitting society through its cultivation of social intelligence (e.g., Oatley, 1999; 
Pinker, 2011, Zunshine, 2006) and opportunities to explore social and moral themes (Hakemulder, 
2000; Nussbaum, 1990, 1995). These ideas have received attention in experimental psychology 
over the past decade, though research findings have been mixed and this may be due, in part, to the 
challenges of operationalizing empathy and selecting fiction stimuli. In this article, I introduce the 
relationship between fiction and empathic abilities, and provide a brief overview of current psy- 
chological research and its limitations. Drawing on the extended mind thesis (Clark and Chalmers, 
1998) and Material Engagement Theory (Malafouris, 2013), I frame both fiction-engagement and 
empathising as partly externalised processes and use this framing to make general suggestions for 
future experimental approaches to fiction effects on empathy. 


Fiction and Social Life 


Fiction, as opposed to nonfiction and other leisure pursuits, may have social and even societal 
benefits. This is due to its inherently social focus (Mar and Oatley, 2008; Oatley, 1999): fictional 
characters are usually human, or they bear psychological resemblance to humans, with non-human 
protagonists imbued with humanlike traits and involved in human social themes (e.g., Mar, 2009). 
Whilst nonfiction narratives can also present social events, fiction, which is understood to be 
nonveridical, provides the opportunity to explore them without real-world obligations or conse- 
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quences (Keen, 2007). Where nonfiction tends to foreground information, fiction is concerned with 
possibility, characters and their emotions (Oatley, 1999), and so fiction readers can gain a level of 
insight into the private thoughts and emotional lives of others usually unavailable in nonfiction prose 
(e.g., journalism, biography and history; Niinning, 2014). 

The more “social” its content, the more people appear to enjoy fiction: Longstanding works such 
as Shakespeare’s plays contain interrelated networks of characters—which reflect the size and struc- 
tures of most human social networks—navigating social themes. Love and status, for example, are 
reflected in Shakespeare’s comedy and tragedy genres, respectively, at their highest stakes (Nettle, 
2005). Children show preferences for stories that contain people rather than objects, particularly 
those which include descriptions of characters’ mental states (Barnes and Bloom, 2014). Apprecia- 
tion of fictional narratives and the development of children’s social acumen may be linked, as the 
understanding that others have different perspectives to one’s own known as “Theory of Mind” 
(ToM; Premack and Woodruff, 1978), typically develops around age four (Wimmer and Perner, 
1983), along with the tendency to become interested in fairytales, and may aid reading comprehen- 
sion (Astington, 1990; Dore et al., 2018). 

ToM can be characterised as a component of “cognitive empathy” (interpreting others’ mental 
states including thoughts, beliefs, motivations and feelings) which, in turn, is contrasted with “affec- 
tive empathy” (sharing the emotions of a particular target; Cuffet al., 2016). | use “empathy” as the 
umbrella term under which cognitive and affective empathic processes sit because its definition 
encompasses cognitive and affective target content, fictional and imagined, as well as real-world 
target agents (Cuffet al., 2016), and also because the term invokes its aesthetic origins: The German 
philosopher Robert Vischer (1873) used “Einfiihlung”, literally translated as “in-feeling” (Waite, 
2012), to refer to the capacity to project feelings of pleasure onto an art piece or object. Subse- 
quently, “empathy” was introduced by the English psychologist Edward Titchener (1909), meaning 
to feel into a person or situation. 

Empathy, then, is not a process, but a state “in which one arrives having undergone those ground- 
ing processes, whatever they may have been” (Smith, 2017, p. 718). In other words, a range of 
processes, such as emotion recognition, contagion, perspective-taking and memory could contrib- 
ute to the state of having empathy (see also, Zaki and Ochsner, 2012). Empathic abilities are gener- 
ally associated with positive interpersonal relationships and prosocial behaviour (Castano, 2012; 
Paal and Bereczkei, 2007), but they vary among neurologically typical adults, and selective deficits 
are characteristic of some developmental differences, clinical and degenerative disorders (e.g., 
Guastella et al., 2013; Poletti et al., 2012). However, these faculties develop through the lifespan (e.g., 
Happé et al., 1998), and can be trained (Teding van Berkhout and Malouff, 2016), and so under- 
standing if and how fiction-engagement might enhance them could have real-world social benefits. 


Research Findings and Challenges 


The most robust psychological evidence of a relationship between fiction and empathy comes 
from positive correlations between familiarity with fiction authors, a proxy measure of fiction- 
reading frequency (Stanovich and West, 1989), and performance on behavioural tests of empathy 
components (for a meta-analysis, see Mumper and Gerrig, 2017). This association appears stronger 
for fiction compared to nonfiction (Mumper and Gerrig, 2017) and to sustain when familiarity with 
nonfiction is statistically controlled for (e.g., Mar et al., 2006; Turner and Vallée-Tourangeau, 
2023). This indicates that the narrative features of fiction enhance empathy above and beyond 
general reading processes. Experimental studies have supported this hypothesized direction of cause, 
revealing a small, immediate and positive effect of fiction-reading on empathy task performance 
(for a meta-analysis, see Dodell-Feder and Tamir, 2018), although this literature is mixed with some 
key results failing to replicate (e.g., Camerer et al., 2018; Panero et al., 2016; Samur et al., 2018; 
though see Kidd and Castano 2017, 201 8a, 2018b; see also, Panero et al., 2017; Van Kujik et al., 2018). 
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Inconsistencies across experimental findings may be attributable to the challenge of working with 
empathy and fiction-engagement, which are both multidimensional constructs (Turner, 2020). 
Studies have used a range of self-report and behavioural tasks to probe empathy processes, including 
perspective-taking (e.g., Mar et al., 2006), emotion recognition (e.g., Pino and Mazza, 2016; Kidd 
and Castano, 2013), and emotional responses to others (e.g., Koopman, 2015; Pino and Mazza, 
2016), as well as associated constructs like sympathy or concern (e.g., Bal and Veltkamp, 2013; Mar 
etal., 2006), and prosocial behaviour (e.g., Koopman, 2015; Johnson, 2012). These processes show 
substantial within- (and between-) person variation (Cox et al., 2012) and they can dissociate (Oakley 
et al., 2016), reflecting the heterogeneity of the empathy construct (Cuffet al., 2016; De Vignemont 
and Singer, 2006). Moreover, the same outcome measures have been used to assess different aspects 
of empathy across studies—reflected in a range of terminology such as “ToM”, “interpersonal sensi- 
tivity” and “mentalizing”—and this raises concerns about construct validity (Black et al., 2021). 

Fictional prose varies by length, theme, genre, narrative and linguistic complexity, narration and 
focalisation, as well proximity to, or mimetic resemblance of, historic or current events. Fictions also 
differ in their ability to imaginatively “transport” readers, a process which appears to moderate some 
narrative effects (e.g., Green and Brock, 2000; Schwerin and Lenhart, 2022). Defining the features 
responsible for fiction effects has thus presented a challenge for researchers. Where studies have 
aimed to do so via experimental comparison (e.g., “literary” versus “popular” genres; Kidd and 
Castano, 2013), random assignment to one of an assortment of texts presents issues of within-group 
heterogeneity even when efforts are made to match text stimuli for length and complexity. This issue 
can be addressed using multilevel statistical models (e.g., Panero et al., 2016) but this has not been 
general practice, which may have led to false positive results (see Judd et al., 2012). The alternative 
is to vary elements of a single stimulus text (e.g., Koopman, 2016), but this strategy reduces 
generalisability, slowing the accumulation of evidence, and while successful replications bolster 
found effects, they do not necessarily shed further light on the mechanisms responsible. 


Reconsidering Reading 


Stories vary not only in their narrative elements, but also in the technologies used to construct and 
present them. This article’s epigraph emphasises the profoundly felt personal and social impact of 
books, yet stories do not only reside on “flat, rigid squares of paper” (Lamott, 1994, p. 15). One could 
be familiar with a wide range of literary works without having read a word of them: stories are 
available on the TV, on film, in the theatre, via interactive and virtual gaming and roleplay, tablets 
and audio devices. Nonetheless, the majority of studies investigating fiction’s effects on empathy 
have focused on reading (Black and Barnes, 2015; Turner and Felisberti, 2018). There are three 
core reasons for this: First, the relationship between stories and empathy has traditionally been the 
subject of literary scholarship where it forms part of a longstanding agenda—rooted in eighteenth 
century distrust of the popularization of books—to understand how literature could influence read- 
ers’ expectations, actions and morals (for an overview, see Keen, 2007). Second, where experimental 
studies have since addressed the question using psychological measures typically designed for devel- 
opmental populations or diagnostic purposes, mixed results have raised doubt about the causal effect. 
Establishing this fundamental link has taken precedence over exploration of potential moderators, 
including medium. Third, the majority of the psychological fiction-effects research is grounded in 
the cognitivist (internalist, representationalist) paradigm, which models cognition as a form of in- 
formation processing in which mental (and neural) structures mediate sensory inputs and behavioural 
outputs (Simon, 1979). Through this lens, the social knowledge or skill acquired through fiction- 
engagement is understood to be consumed, internalised and later applied to real-life scenarios. This 
does not preclude the use of other media stimuli, but emphasis on the manipulation of social content 
“in our heads” (Gallagher, 2013) results in ontological indifference towards the form—and associated 
material properties—through which that content is presented. Using text, which is often the most 
practical approach, is deemed methodologically sufficient. 
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These assumptions warrant reconsideration. First, it is true that reading literature continues to 
account fora substantial portion of many people’s leisure time (e.g., Barnes, 2012); however, more 
people watch television (e.g., in the US, UK, China and Germany; Bureau of Labor Statistics, 2017; 
Seddon, 2011; Statistica, 2022; Stiftung fiir Zukunftsfragen et al., n.d.), and many engage with 
cinema, theatre, radio, audiobooks, and virtual gaming. In order to create a culturally relevant 
account of fiction effects, it is important to address the range of technologies through which fiction 
can be, and is, consumed. Second, robust correlational evidence (Mumper and Gerrig, 2017) and 
positive experimental findings (Dodell-Feder and Tamir, 2018) support the existence of a true effect 
of fiction on empathy. Certainly, it is necessary to confirm and unpack the mechanics of this rela- 
tionship, but emphasis on reading is not a prerequisite. Reading is not the original method of 
engaging with stories nor the most common. Oral storytelling is culturally universal (Brown, 1991), 
has existed since humans developed the capacity for speech (Zipes, 2012), and the earliest stories 
were shared in this way. For example, the Cosmic Hunt myth appears to have been told when there 
was a land bridge connecting the areas that are now Alaska and Russia, dating it between 28,000- 
13,000 BC (Storr, 2019), and Aesop’s Fables were not written down until the 200s BC—three 
centuries after they were created. Spoken word, performance art or radio, therefore, are closer than 
books to the origins of storytelling. 

Finally, most current studies of fiction effects on empathy implicitly cast cognition as the “brain- 
bound affair” (Barona, 2021, p. 138) of generating internal representations from external stimuli. 
ToM, for example, tends to be measured using false-belief tasks, social vignettes or narratives, or via 
emotion recognition tasks (for an overview of behavioural ToM tasks, see Turner and Felisberti, 
2017). These tools are not theoretically neutral but signify a specific perspective which construes 
individuals as having “folk theories” of behaviour that they use to ascribe mental states to others. 
Consider the most commonly used behavioural task in the fiction-empathy field (see Dodell-Feder 
and Tamir’s, 2018, and Mumper and Gerrig’s, 2017, meta-analyses; see also Black et al., 2021): 
Baron-Cohen et al.’s (2001) Reading the Mind in the Eyes Test. This task requires participants to 
interpret the emotions contained within photographs of disembodied eyes and to ascribe appropriate 
verbal labels from selections of terms. Other common measures ask participants to name thoughts or 
emotions based on pictures or vignettes (e.g., Emotion Attribution Task [Blair and Cipolotti, 2000]; 
Yoni Task [Shamay-Tsoory and Aharon Peretz, 2007]), or to reflect on, and express via self-report, 
their general tendencies to appreciate others’ mental experiences (e.g., dimensions of the Interper- 
sonal Reactivity Index [Davis, 1983]; Toronto Empathy Questionnaire [Koopman, 2015]). Thus, 
the most commonly used tasks have been based on the presumption of a detached observer who uses 
mental representations to consciously reflect on the mental states stored in other people’s heads. 

The rise in “4E” approaches offers an alternative ontology of mind, where cognition is characterised 
notas taking place solely “north-of-neck” (Fodor, 1999, p. 98), but as embedded, enacted, embodied or 
extended via processes and structures outside the head (for an overview, see Carney, 2020). Propo- 
nents of 4E approaches argue that the cognitive processes studied in modern cognitive science are 
dependent on the agent’s body, their environment, and interactions between the two, though they 
vary in how these integrate (Newen et al., 2018). Take, for example, numerical cognition: counting 
need not rely on internal linguistic representations (Clark, 2006) when natural numbers can be 
found, used and manipulated on bodies (fingers and so on, e.g., Saxe, 1981) and in the environment 
(e.g., tokens and tallies; De Cruz, 2008). Although they are unified in opposition to internalist, brain- 
bound perspectives, there are conflicts within 4E approaches (Shapiro, 2010). For example, embedded 
theories retain an emphasis on internal, representational structures that contain knowledge about the 
world (Kiverstein, 2018), whereas the extended theories based on Clark and Chalmers’s (1998) ex- 
tended mind (EM) thesis argue that, due to environmental resources taking an active role in cognitive 
processes (e.g., tools and technologies such as diaries and computers), they can, in some conditions, be 
considered constituents of those processes (Kiverstein, 2018; Malafouris and Renfrew, 2010). 
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Social cognition is the system of mechanisms and processes that enable humans to make sense of 
social information and behave appropriately in specific social contexts (Shany-Ur and Rankin, 
2014), and so the environment is fundamentally important to accounts of social cognitive process- 
ing. Works of fiction, as inherently social and modally diverse, represent information to be pro- 
cessed using social cognitive apparatus as well as tools, situated within environments, that can be 
considered constituents in those processes. In the following sections I draw on extended approaches 
in order to outline how the properties of a given fiction presentation are integral to the emergence 
of the empathic processes involved in its understanding, and describe how fiction-engagement and 
empathy processes may continue to impact each other over time at the cultural level. 


Fiction and Empathy in the Material World 


Traditionally, theory-based accounts of empathic abilities (e.g., ToM; see Coll et al., 2017) have 
been contrasted with simulation accounts (“Theory-Theory” versus “Simulation Theory”); the lat- 
ter being the idea that observers use the same mental apparatus when interpreting the thoughts and 
feelings of a target person as that person uses when having those thoughts and feelings (e.g., Coplan, 
2011; Goldman, 1995, 2006). Despite often being pitted against one another, these accounts are not 
mutually exclusive, and social cognitive processes have been modelled using both domains (sup- 
ported by neuroscientific evidence, Zaki and Ochsner, 2012; Goldman’s, 2006, simulation account 
is also integrative, but with simulation at the forefront). This dual-process approach is reflected in 
Mar’s (2018) proposal that fiction-engagement may enhance empathy through two routes: the 
accumulation of social knowledge via the social information presented (“content”) and the recruit- 
ment and honing of mental apparatus used in real life social scenarios (“process”), with the majority 
of studies revealing effects along the latter route.' For example, Tamir et al. (2016) found that brain 
activity in the dorsomedial prefrontal cortex, a subnetwork of the default network—the latter known 
to be involved in simulation of spaces, scenes and mental states—mediated the positive relationship 
between fiction exposure and performance on a ToM task. This builds on the substantial body of 
work revealing perceptual and motor simulation processes involved in the interpretation of lan- 
guage (e.g., Speer at al., 2009; for an overview, see Bergen, 2012). 

In social cognition, too, Simulation Theory has been endorsed by the discovery of mirror neurons 
that enable automatic mimicry at the neural level (Rizzolatti et al., 1996). Emphasis on neuronal 
activity can perpetuate an all-in-the-head ontology of empathy, yet empathy represents an embod- 
ied phenomenon since mental states are enacted via the animate body (via facial expressions and 
gesture) and, as they are directly perceptible, they do not need to be interpreted using folk theory 
(Krueger, 2009). Empathy is also situated within specific social interactions, and it is extended, 
because when sharing in another person’s experience, particularly if mirror neurons are “coupled” 
via simulation (lacoboni, 2008, p. 265), the perceiver exploits the part of the environment that is the 
other person. Thus, the individual is extended socially (the cognitive process is “distributed” across 
individuals; Flor and Hutchins, 1991), having evolved to be “plug-compatible” (Kosslyn, 2007, p. 
547) with other humans. This is not to say that traditional perspectives on social cognition no longer 
have a place. Rather, 4E approaches can enrich these “representation-hungry” accounts (Herschbach, 
2018, p. 524) in that cognitive events, such as the interpretation of a target’s emotional state, are seen 
as extended and processual, involving the brain, but also the body and its surrounding context 
(Krueger, 2009). This model moves intersubjectivity, at least in part, out of the head and into the 
systems and structures of the social environment. 

Empathy, then, is extended through other people (day-to-day empathy), as well as through fic- 
tional tools (fictional empathy); works of fiction are part of the social environment and they are also 
tools for presenting social life. From a cognitivist perspective, fictions are products of a unidirec- 
tional, causal process wherein thoughts or mental representations flow out of the mind of writer 
and are put onto paper via the writer’s hand; the mind is sequestered away from the body which, in 
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turn, is detached from pen, ink and paper (or computer keyboard; Bernini, 2014). This leaves little 
room for the kinetic and material dimensions of the “doing” of writing (Freiman, 2015; see also, 
Booker Prize nominee Alan Garner, 2022, describing his preference for the “mysterious” kinetic 
and intuitive experience of writing rather than typing), for writing as thinking (Menary, 2007; 
Oatley and Djikic, 2008) or as “thought in action” (Menary, 2007, p. 630); rather, penned or typed 
language simply functions as external storage for internal ideas. If story-production involves a one- 
way route from brain to page, one could not say “the book wrote itself”, or “the characters took on 
lives of their own” (see Taylor et al., 2003), or even “it didn’t make sense until I read it back”. There 
can be no dialogue between the characters and their writer who feels compelled to create particular 
outcomes for them and empathises with them when reading back, acting simultaneously as author 
and reader, nor can the experience of developing ideas through brainstorming or redrafting be 
readily accounted for (e.g., Bernini, 2014; Clark and Chalmers, 1998; Freiman, 2015). This yields a 
static object incapable of surprising the author, of which interpretation constitutes an entirely sepa- 
rate act. On the other hand, considering story-making as a process of externalised cognition posi- 
tions both the author’s mind and external writing materials, including language (Clark, 2008), and 
technology (Bernini, 2014), as constituents of the cognitive ecosystem (Hutchins, 2010) from which 
the fictional narrative emerges (Clark and Chalmers, 1998), and invites the reader into the equation. 

The theory that fiction can influence real-world empathy for others is based on the notion of a 
reader who actively makes sense of the story (e.g., Barthes, 1967/1977) and so who, through reading, 
has a functional role in its creation (I use “reading” as shorthand for the consumption of fiction ina 
given format). The extent to which imaginative effort, as opposed to more passive or immersive 
engagement, underlies fiction effects on empathy, remains unclear, with more research needed (see 
Hakemulder, 2000; Kidd and Castano, 2013, cf. Turner and Vallée-Tourangeau, 2020). However, 
the implication is that just as empathy emerges from interaction between two or more systems (those 
of empathiser and target), the meaning of a literary narrative is distributed across (at least) two 
minds—those of author and reader—and unfolds amid the interplay of the systems surrounding each. 

Material Engagement Theory (MET; Malafouris, 2013) sheds light on the nature of these systems. 
Like EM, this cognitive archaeological approach moves beyond internal mentation to describe and 
examine the “middle space where brain, body, and culture are conflated” (Malafouris, 2020, p.3). As 
with EM, in MET, external artifacts (as well as bodily and cultural things) are conceived as cognitive 
constituents. In his challenge to anthropocentric definitions of material agency, where material 
objects are only considered active in relation to human use of them, Malafouris’s (2008, 2013) MET 
account distinguishes between having a sense of agency, which may be a solely human phenom- 
enon, and agency itself, which is not (Barona, 2021).? To borrow Malafouris’s example, a clay pot is 
not simply the product of the potter’s mental idea for a pot with which the clay subsequently com- 
plied. Although the potter may have a sense of their own agency in the process, the pot was formed 
not by the potter’s impositions on the clay but via dynamic (and potentially equitable; Clark, 2007) 
interplay between the potter’s body, the wheel, and the clay’s affordances (Gibson, 1979). There is an 
implied symmetry (Latour, 1999) between people and things (Malafouris, 2020), where both have 
agency, with things at once external to the body but internal to the cognitive ecosystem. 

Making fiction, technologies such as pen, paper, computer keyboard, as well as the materiality of 
words (printed or sounded out) are the things internal to the process while being external to the 
writer’s body. Just as the potter has a sense of authorship as the creator of the finished pot, the writer 
feels that they have authored the written story, though it is actually a process of co-creation between 
people and materials (Malafouris, 2013), emerging via “inextricable tangles of feedback, feedforward 
and feedaround loops that promiscuously criss-cross the boundaries of brain, body and the world” 
(Clark, 2012, p. 277). Depending on medium, authorship (and sense of authorship) may be distrib- 
uted across a range of creative units. In film, for example, authorship is routinely extended across 
translators, screenwriters, producers, directors, actors, musicians, make-up artists, set designers, edi- 
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tors and so on. Each unit or department, drawing on its own specific equipment and materials, 
contributes to the larger project of the film as a whole. The “language” of the story refers both to the 
words used to convey its narrative and to the semiotic opportunities afforded by the medium 
through which it is presented (e.g., cinematic language). 

When interpreting this language, the reader brings their unique, inter- and intra-cultural experi- 
ences of similar environments, including encounters with the language of the format, such as expe- 
rience with cinematic or theatrical devices, particular themes and tropes, as well as the social envi- 
ronment associated with the modality. To borrow from Gallagher and Ransom’s (2016) application 
of MET to joint attention, a social interaction in which individuals coordinate their actions, involv- 
ing the social cognitive ability to move beyond egocentric perspectives on the world and meet 
another person in a shared cognitive space, the reader’s sensemaking is “extended across brains- 
bodies-agents-environments, in ways that incorporate relevant (and potentially unique) background 
components” (p. 344). A tool’s usage or “meaning” or “what’s-it-for-ness” is not stable and objective 
but changes with each new user, or with the same user over time, and this is impacted by the present 
environment as well as experience of previous environments. In the same way, the meaning of a 
story is not stable but varies with the affordances and environment of each presentation of it both 
within and between readers. 


Originations and Interpretations 


If fiction sensemaking emerges from interactions between story authors and story readers (Popova, 
2014), with the reader both impacted by the story and active in the construction of its meaning, then 
the process of reading represents a spiralling system of origination and sensemaking or interpreta- 
tion. In this way, the continuous creation of any work of art is distributed along its path of life, 
rendering unclear whether it is the artist or the audience who is creating the work (Latour, 2013). 
This dynamic process occurs across several levels. At the individual level (insofar as any aspect of the 
process can be thought of as concerning individuals disentangled from sociocultural contexts), read- 
ers vary in their style of engagement; for example, by taking protagonist versus eyewitness perspec- 
tives regardless of whether the narrative is first- or third-person (Hartung et al., 2017). Additionally, 
bringing new life history to each reading means they experience the story differently every time. At 
the social level, readers may engage in conversation and commentary which layers meaning onto 
the “finished” fiction product, and fiction has recently become more interactive due to the internet 
facilitating discussion amongst audiences and critics, fan theories and fan fiction creations (see Rose, 
2011). At this level, interpretations of a work of fiction can diverge a great deal: “the more we 
interpret it the more we unfold the multiplicity of those who love it as well as the multiplicity of what 
they love in it” (Latour, 2013, p. 241). 

At the cultural level, each subsequent iteration of a story contributes to its cultural manifestation 
(and the internet has made stories evermore transmissible across cultures). This layers onto the story’s 
next telling—be that in conversation, via commentary or a new format—as well as on “new” fiction 
productions. For example, linguistic intensifiers such as “really”, “very” and “so” in the TV sitcom 
Friends (Bright et al., 1994-2004) reflected but may also have innovated language trends (Tagliamonte 
and Roberts, 2005), and the durability of Shakespeare’s works has resulted in many of his phrases 
appearing in contemporary parlance. In the former, the (untrue) reference to lobsters as animals that 
mate for life (“he’s her lobster”) may have influenced the darkly comic absurdist film The Lobster 
(Lanthimos, 2015)—which features an institution that requires single people to meet romantic part- 
ners or else be permanently transformed into an animal of their choosing—or audiences’ interpreta- 
tions of it at least. Dickens’s Oliver Twist raised awareness of the cruelty of the workhouse system in 
Victorian England. Originally published in monthly instalments, a large portion of the chapter 
following Oliver’s request for more food describes the punishments visited upon him for doing so 
(Richardson, 2012). Much of this tends to be omitted from contemporary versions of the story, 
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perhaps because the workhouse system no longer exists and modern audiences do not have so strong 
an appetite for the visceral experience of it afforded by film or TV remakes. Pinker (2011) has 
argued that the perspective-taking experience afforded by books—which became widespread fol- 
lowing the development of the mechanical printing press (in the mid-1400s; Lagerfeld, 1986)— 
reduced public enjoyment in witnessing the suffering of others (e.g., attending public corporeal 
punishment). Thus, the general effects of fiction on empathy can be seen to have directly influenced 
the ways in which such stories get retold. Whilst Friends, Shakespeare and Dickens represent par- 
ticularly influential examples, the cultural lives of all works of fiction are constructed through ongo- 
ing spirals of making and sensemaking which dynamically intersect the levels of individual, society 
and culture. 


Cognition and Culture 


Engaging with falseness in fiction is a common phenomenon but it is also a strange one: stories 
communicate cultural values, which impact cultural practices, but while people are intensely interested 
in accuracy when information is intended to be truthful, they selectively suspend disbelief for fiction 
(Tooby and Cosmides, 2001). Difficulties interpreting pretence in some populations (e.g., autism; 
Jarrold, 2003) indicates that this capacity is not a by-product of general intelligence but arises from 
a specialized subsystem built into the human cognitive architecture (Tooby and Cosmides, 2001). 

If fiction-engagement and empathy are served by the same social cognitive mechanisms, includ- 
ing mentalizing, emotion perception and response, then fiction can be seen to shape empathy and 
empathy to shape fiction as part of the dynamic process of origination and interpretation (or making 
and sensemaking) characterised above. The MET perspective foregrounds the role of material 
culture in this fiction-empathy loop. According to MET, external materials do not simply scaffold 
the ways people think, they also shape them. To revisit the numerical cognition example from 
earlier, the archaeological record indicates that numeracy co-evolved with the making of clay 
tokens used to perform accounting tasks (Malafouris, 2013), and addition appears to have emerged 
cross-culturally before subtraction and other operations, perhaps because adding notches to a tally 
stick is easier than erasing them (Overmann, 2016). Humans do not create a tool or a work of art via 
a one-way causal process; “we make things which in turn make us” (Ihde and Malafouris, 2019, p. 
195). This macro perspective, which explains how tool use can impact cognition transactionally as 
well as cross-culturally and over time, resonates with the central assumption of fiction-empathy 
research that both real and fictional social information is understood using the same cognitive 
mechanisms (Gerrig, 1993; see also, Bergen, 2012), and that the impetus to experience empathy and 
explore social stories hones the processes which support their understanding (though this is usually 
described in terms of knowledge acquisition and internal mentation; Mar, 2018; Mar and Oatley, 
2008; Oatley, 1999, 2011; Oatley and Djikic, 2017; cf. Currie, 2003, and Keen, 2007). 

New technologies are not conjured by authors who require them to tell particular stories. Rather, 
specific technological affordances mediate the creation and interpretation of stories in ways which 
impact their real-world effects on empathy and, in turn, future iterations of those stories. For ex- 
ample, the advent of private, leisure-reading from the 1400s—late 1700s that followed the develop- 
ment of the mechanical printing press (Lagerfeld, 1986), enabled mass access to the personal lives of 
distant characters. The popularity of stories consequently expediated the evolution of a range of 
mass storytelling technologies, each entailing the engagement of different empathic processes. De- 
spite the subsequent proliferation of radio, television and the internet, print literature (along with 
later technologies) has sustained (or adapted; Scolari, 2013), indicating that it affords something its 
counterparts do not; it is functionally unique. As McLuhan (1964/1994) put it, “the medium is the 
message”: medium is fundamental to the way content is experienced. The medium is also the mas- 
sage (McLuhan et al., 1967/2008)—it affects the whole sensorium (this update amounted from an 
error in typesetting the original phrase for the new publication which McLuhan liked and decided 
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to keep; McLuhan, n. d.). As fictional content presented via different technologies is perceived and 
processed through the body in different ways, material culture is essential to the way it is engaged 
with, understood and ultimately reapplied. 


Extending Fiction-Empathy Research 


If we accept Malafouris’s proposal that minds and things are not only causally linked but are 
“constitutively interdependent” (one cannot exist without the other; Malafouris, 2013, p. 77), as well 
as the theory that fiction-engagement and empathising are consubstantial, involving the same cog- 
nitive apparatus, then we must view the material affordances of fiction technologies as constituent in 
the emergence and evolution of empathy. This is not simply an abstract, theoretical consideration: 
because empathy is multidimensional, the mode through which a story is presented may have 
specific and unique effects on different and sometimes dissociable processes. 

Consider, for example, variations in the presentation of Lady Macbeth’s descent into madness 
afforded by different fiction technologies. The radio actor delivers lines with a trembling voice, 
whereas film enables close-ups of a tearful face, atmospheric scenography, music; perhaps even an 
informative flashback. At the theatre, the actor’s voice is projected into the auditorium and this 
technique necessitates a level of physical effort and accompanying bodily expression that film acting 
does not usually require. Meanwhile, when reading the text in a volume of The Complete Works of 
Shakespeare, imagination helps to set the scene, adjusted by prior knowledge (including of the story 
and its many iterations), and constrained by the reader’s present environment. Reading, listening or 
viewing will engage the “grounding processes” (Smith, 2017, p. 718)—emotion perception, memory, 
simulation and mental representation, for example—that contribute to empathy, to different de- 
grees. Thus, mental representation alone does not account for the complexities of empathy in fic- 
tion-engagement; rather, it has a greater or lesser role depending on modality, with certain tech- 
nologies facilitating direct perception of social content in ways that more closely reflect real-world 
social processing. 

Moreover, the reader, listener or (at-home) film viewer has the option to move forward and 
backward between scenes, revisit key moments or background information, skim over or skip gory 
sections, and check how the story concludes. Recalling empathy’s origin as an aesthetic experience, 
the tool becomes something that is felt into and experienced as responsive to the agent’s explorations 
(Chemero, 2016). This level of interaction would be impossible in the theatre or cinema, although 
these modes may incorporate other devices to impact sensemaking (e.g., “easter eggs”—hidden 
elements which call-back or foreshadow parts of the story or reference other stories). New theatre 
technologies, for example, have increased options for cueing action (e.g., stagehands wearing head- 
sets), enabling productions to incorporate complex and multimedia design elements, interactive 
costumes, performer-controlled lighting and sound, audience participation and multi-sensory 
immersive experiences (Nicholas et al., 2021). Advances in interactive viewing technologies have 
recently enabled mass engagement with do-it-yourself film plotlines—in 2018, streaming service 
Netflix released its first interactive film, Black Mirror; Bandersnatch (McLean and Slade, 2018)— 
facilitating risk-free exploration of potential outcomes to a range of fictional social scenarios. 

Active “writerly” engagement (Barthes, 1967/1977) may not only be a function of some fiction 
formats, but may be required for empathic processes to be meaningfully impacted (Hakemulder, 
2000; Kidd and Castano, 2013; Zunshine, 2006). It may be that the relative visual and auditory 
poverty of books entails empathic faculties to be engaged through simulation to a greater degree than 
film, for example, through which some mental states can be directly perceived. On the other hand, 
experience with close-ups in film might support facial emotion recognition or emotion sharing, par- 
ticularly when paired with a congruent soundtrack. Any mode which allows the reader or viewer to 
refer back to important contextual information might support accuracy or efficiency in interpretating 
complex beliefs and intentions. The uniquely social, collaborative and synergistic engagement op- 
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portunities of immersive theatre and virtual reality might engage these faculties further, as partici- 
pants become acutely aware of their own agency and impact on the narrative world (e.g., Turner 
and Kasperczyk, 2022). Here, the “exchange of one’s own reality for the sensations of another takes 
to its furthest logical extension the fusing with another object that aesthetics’ Einfiihlung set out to 
describe” (Keen, 2007, p. 39). These questions can only be explored through examination of the 
varying textures, social and material affordances of different modes of fiction-engagement. 

While most fiction-empathy research has used written text as stimuli, some studies have examined 
the effects of other presentations of fiction (e.g., Black and Barnes, 2015; Mar et al., 2010; Turner 
and Felsiberti, 2018) and yielded promising results (cf. De Mulder et al., 2022). How far different 
technologies correspond with different empathy components has yet to be identified, however, as 
well as strong evidence of causation. This could be addressed by examining the effects ofa single, 
selected technology ona selected empathy component or components. To some degree, this ap- 
proach is already being taken, albeit for different reasons: in some studies, text is presented on paper 
and in others on screen, a practical consideration which, despite evidence that this likely results in 
processing differences (with documented effects on absorption and comprehension; Mangen and 
Kuiken, 2014; Singer Trackhman et al., 2019), rarely features in theoretical discussions. 

A more ambitious approach would involve comparing effects of different presentations of the 
same story on a selected empathy component or components; for example, comparing reading a 
story to listening, to viewing a film version or live performance of it, to participating in gaming or 
interactive performance versions. Experiments could measure empathic accuracy or efficiency, 
while incorporating different target content via visual, auditory, and narrative tasks (for an over- 
view, see Turner & Felisberti, 2017), because identifying the particular domains impacted within 
participants is essential to understanding how far specific technological aftordances contribute to 
specific empathic processes. Developing this line of enquiry, “active” versus “passive” engagement 
styles could be compared between technologies, by contrasting theatre, gaming or film, where visual 
and auditory cues are provided, to reading, where they are intuited. This could ultimately be 
combined with within-technology comparisons of complex or self-guided versus predictable 
plotlines, the presence versus absence of specific stylistic features (e.g., Koopman, 2016), or manipu- 
lation of engagement levels (e.g., Turner & Valleé-Tourangeau, 2020). These approaches would 
enable assessment of the transferable effects of social processing aftorded by different fiction tech- 
nologies. However, larger participant numbers would be required to reduce statistical noise, and 
researchers creating heterogenous conditions (e.g., by using a range of texts) should account for the 
random effects of stimuli via mixed effects analyses. 

The field of experimental research in fiction effects on empathy has inherited several method- 
ological challenges. Many of the measures employed were originally designed to detect deficits 
rather than natural variation within and between individuals (Turner & Felisberti, 2017), and the 
lack of uniformity across measures, as well as the range of fiction stimuli used between and within 
experiments, has made it difficult to confidently infer fiction’s causal effects on empathy. In the real 
world, people are exposed to fiction via different media over time, whereas most experimental 
protocol has involved examining empathy levels immediately after exposure to short excerpts of 
fictional prose (Mar, 2018; Quinlan et al., 2023). Moving forward, researchers may conceive en- 
tirely new, de-individualised methods of investigation which do not demarcate thought from em- 
bodied activity (see Malafouris, 2013) or from its social, cultural and material context. Such ap- 
proaches could incorporate observational methods and yield rich, longitudinal data aimed at cap- 
turing situational effects and establishing their durability, as well as long-term interactions between 
patterns of fiction-engagement and empathy. In the meantime, the suggestions offered here are not 
aimed at further complicating data collection, but at a reorientation toward material technological, 
rather than abstract stylistic fiction comparisons. This approach has implications at the individual 
level, with potential application to specific areas of social cognitive skills development, and at the 
sociocultural level, where it may offer insight into the gradual expansion of empathy for others 
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beyond the immediate ingroup (Pinker, 2011; Singer, 1981/2011), as well as observed declines in 
some empathy components in certain populations (e.g., American university students; Konrath et 
al., 2011; see also, Zaki, 2019). Bringing the extended architecture of empathy and fiction-engage- 
ment into focus offers a framework for understanding the role of fiction technologies in shaping, and 
being shaped by, the multidimensional empathy skillset. 


Conclusion 


Traditional information-processing accounts construe cognition as an intracranial affair (e.¢., 
Adams and Aizawa, 2008), and social cognition as an inter-intracranial one, realized via brain 
processes (Newen et al., 2018). EM (alongside other 4E approaches) and MET have offered an 
alternative ontology of mind, arguing that it can be extended beyond the brain, via interaction 
between brains, bodies and environments. Fiction-engagement and empathising are cognitive acts 
that occur, at least in part, outside of people’s heads. However, extant fiction-empathy research in 
Psychology has been based on traditional cognitivist models which conceive readers as gaining 
social skills through the acquisition of social knowledge contained in fiction or via mental simulation 
reliant on internal representations. MET explains how, as humans, we define and make ourselves 
through technologies and tools. This is also true of fictions, through which human social life is both 
presented and learned from via ongoing interaction between socially, culturally and environmen- 
tally situated systems of origination and interpretation. Inviting the environment of the reader or 
sensemaker into the fiction effects equation via systematic study of fiction technologies would 
provide a more comprehensive and culturally relevant account of the fiction landscape, and the 
opportunity to model the architecture of the evolving fiction-empathy relationship. 
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Notes 


' For summaries of this and other theories of narrative effects in fiction and nonfiction, see Green et al. (2020). 

? Latour (1992, p. 241) uses the term “actant” to denote the things which become active through some form 
of doing, and in the field of narratology, the same term refers to roles such as “hero” or “villain” on whom 
astory’s structure relies (Greimas, 1973/198 7). 
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Of Mice and Meaning: Multidisciplinary 
Perspectives on the Interconnectedness of Pain, 
Stress, and Suffering 


CLAIRE WOODWARD & TAYLOR WOODWARD 


Abstract: We document definitions of and relationships between pain, stress, and suffering while 
also reflecting on our own diverse training and experiences. Recognizing a range of causes and 
interpretations, we differentiate between maladaptive and adaptive forms of pain, stress, and suffer- 
ing. Measures and identifiers of pain often rely on quantifiable measures, while suffering demands a 
greater attention to perceptions of self-understanding. Engaging with pain, stress, and suffering 
means considering how the mind and brain work together in processing and articulating moments 
of physical agony, heightened anxiety, and unbridled grief, as well as the stories we tell ourselves 
about these experiences. 


Keywords: pain, stress, suffering, interdisciplinary 


Postulating Pain 


“All becoming and growing, all that guarantees the future, postulates pain.” 
Friedrich Nietzsche 


p= and suffering are an inherent part of the human experience. As such, humans have constantly 
searched for answers to explain suffering and turn pain into growth and meaning. Narratives 
and art depicting the human emotion of pain can evoke feelings of connection and empathy among 
individual hearts and minds as well as insights into the sublime as interpreted by Immanuel Kant, 
though suffering can also decrease quality of life. There are then both adaptive (beneficial) and 
maladaptive (adverse) behaviors that arise from pain and stress with suffering, which complicates 
the way we study and attempt to limit their negative effects on human well-being. Our own schol- 
arship is motivated by our desire to limit maladaptive pain, stress, and suffering. 

Weare siblings and scholars from seemingly disparate disciplines (one in humanities and one in 
neurobiology), yet we are both engaged in work to understand—and ultimately alleviate—various 
facets of pain and suffering. We currently work in two different laboratories at the same institution 
(Indiana University). Taylor works in a lab with mice and Claire works in a lab on storytelling. Our 
scholarship includes interpreting reactions to pain, stress, and suffering, both felt and observed, 
behavioral and aesthetic. Such observations provide further insights into understanding how the 
mind and the brain process pain and suffering. 

Scientists and humanities scholars examining pain, stress, and suffering act as both storytellers and 
data miners through their respective methodologies. In this article, we reflect on our academic 
training while also connecting varying interpretations of pain, stress, and suffering to explore the 
relationship between the three. Definitions of each category are important because they affect 
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artistic representations as well as interventions, which in turn influences the ways in which we try to 
minimize unnecessary pain, stress, and suffering. On one hand, Batson (2009) has highlighted the 
paradox that there is a near universal agreement on the importance of empathy, yet on the other 
hand there are vast disagreements over why empathy is important as well as its effects, origins, and 
basic definitions (see also Decety & Ickes 2011). 

In similar fashion, we suggest an assumed congruency in that identifying, measuring, and finding 
effective interventions for pain and suffering are critical in decreasing harmful suffering, although 
differences exist across definitions, methodologies, and expected outcomes. Kanov (2020) has of- 
fered definitions of suffering as well as arguments as to why suffering matters, and we add to this 
work by putting pain, stress, and suffering into conversation with each other. Both pain and stress are 
often experienced as external forces (“my arm hurts me” or “my job is stressful”) whereas suffering is 
frequently expressed as a part of self-understanding (“I suffer” as the self actively performs the verb 
“suffer”). By categorizing and differentiating types of pain, stress, and suffering as well as sharing our 
own experiences, we provide insights as to how the mind and the brain work in tandem and how 
suffering in particular affects one’s sense of self: This partnership of mind and brain controls both the 
way that we experience and respond to unpleasant and potentially debilitating stimuli, as well as the 
way we measure and represent these experiences. 

While other scholars have theorized about the differences between pain and suffering, this article 
offers the inclusion of stress within this relational scheme. Though many scholars from the humani- 
ties borrow from the sciences and vice-versa, this is the first type of direct comparison to our knowl- 
edge between two scholars of such diverse backgrounds in collaborating on pain, stress, suffering, and 
their interconnectedness. In doing this, we attempt to open a conversation on the following questions: 


- Why is it important to define and recognize the interconnectedness between pain, stress, and suffer- 
ing from diverse disciplines? 

- What are the potential benefits and harms for individuals and societies that stem from pain, stress, and 
suffering? 

- How will considering both the potential adaptive and maladaptive components of pain, stress, and 
suffering serve to guide future interventions to decrease needless suffering? 


Claire’s Background 


My academic journey began as an undergraduate studying history and German. I found myself 
drawn to the social, political, and moral complexities of the past and concerned by the extent of 
human suffering that shapes history. I entered a PhD program in Germanic Studies and was imme- 
diately overwhelmed by the task of making meaning from terms like “mimesis,” “Hegelian,” “po- 
lemic,” and “ontological,” even after looking up their respective definitions. Yet somehow, I was still 
surprised when I needed to spend thirty minutes explaining what “aesthetics” meant to Taylor as we 
worked on this manuscript. Much of my current research stems from projects in Indiana University’s 
Experimental Humanities Lab, directed by Fritz Breithaupt. These projects rely on collaboration 
across disciplines regarding interests, including empathy, narrative retellings, interactive fiction, 
violence, and social polarization. Though our lab produces theoretical scholarship (see for example 
Hiskes et al. 2022), most of our research is experimental (see for example Lagrange et al 2019), with 
data coming from survey responses to narratives we strategically manipulate. 

Ona more specific level, | examine cognitive and affective models of suffering in German litera- 
ture and film. However, my scholarship has also felt very personal. As Cynthia Wallace has noted 
regarding her own work in literary studies, “We learn about suffering from our reading, and we 
learn about reading from our suffering” (2016, 231). After an emergency cesarean section in 2021, 
[had to rate my pain ona scale from 1-10 for a nurse every hour for the two days I spent recovering 
in the hospital. Each time, I agonized over what the difference between a 4 and 5 might be. When I 
delayed a narcotic for treating my pain because I had been feeling pretty good—maybe a 3—I 
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suddenly found myself sobbing in severe abdominal pain. A nurse came in and asked me my pain 
levels, to which I cried out, “Maybe a 62” I hated admitting to being over a 5 but also felt like my tears 
betrayed a pain I did not want to acknowledge. My cognitive brain told me I was fine while my body 
viscerally rebelled. Admittedly, I was in no condition to narrate any immediate postpartum pain 
beyond a numbered scale, but I remember feeling the difficulty in quantifying my pain in order for 
a nurse to understand my level of discomfort and thus provide the best care. The ability to address 
pain in clinical settings comes largely from the ability to discern it in others, and thus both quantita- 
tive data and interpretative skills prove important in relieving pain. 

In early 2023, I once again found myself in the hospital, this time with a late miscarriage at around 
16 gestational weeks. The medical staff was still concerned with my ratings of pain as I progressed 
through labor, delivered our tiny baby, and then had to have a D&C to remove a stubborn placenta. 
In addition to tracking my pain assessments, the hospital staff attempted to address the emotional 
suffering I was experiencing. A grief counselor visited me and my husband a few hours after the 
delivery and offered to talk about our loss, though we were beyond words in our raw mourning. The 
counselor provided us with information on other resources such as therapy and support groups 
around late pregnancy losses, opportunities to narrativize our experiences as well as sustain others in 
their own stories. However, during those hours at the hospital, I could only think of my own inability 
to ever articulate or represent such terrible loss in any form. When asked to do so, I could offer 
quantitative estimates of physical pain. But suffering this kind of loss was (and is) a process, a devel- 
oping narrative that would not be neatly resolved in the same way medication eased my bodily pain. 
This type of suffering will forever be tied with an understanding of myself. 


Taylor’s Background 


As an undergraduate, I gravitated toward studying neuroscience because the child in me wanted 
to bring my fascination with taking things apart to life. I wanted to learn how to ‘take apart’ the brain 
with the goal of understanding and potentially ameliorating the mental health challenges that I and 
so many of my friends and family faced. I sought to understand what drove many people to purpose- 
fully seek out addictive drugs despite the blatantly negative consequences. I wanted to know the 
mechanisms of the antidepressants I took for my own battle with anxiety and depression. To pursue 
these interests, I joined a research lab and spent my time investigating how alcohol (ethanol) affects 
the electrical activity of mouse neurons in a brain region associated with reward and motivation. 

After receiving rejections from all the neuroscience PhD programs to which I applied (my science 
marks were ‘quite poor’ as one program put it), | was told that working as a technician would 
increase my chances of acceptance the next time I applied. I then worked for two years as a research 
assistant in a neuroscience lab at the University of Denver-Anschutz Medical Campus. As I con- 
ducted research and interacted with individuals from the campus, I gained more awareness of 
different scientific philosophies and approaches regarding health and disease within the biomedical 
sciences. On one hand, there are those engaged in clinical research, characterizing diseases and 
disorders in human populations and detailing the effects of specific interventions (both drugs and 
non-pharmaceutical treatments) on their subjects. On the other hand, there are the ‘basic’ scientists, 
those who often work with rodents, cells, and chemicals, to further our understanding of basic 
biological processes that govern life. However, one who attempts to approach real-world problems 
with basic science tools and techniques is often referred to as a ‘preclinical’ or ‘translational’ scientist. 
I became enamored with the idea that the research I performed in a more ‘basic’ lab setting with 
rodents could translate to the world as a whole and benefit my friends and family dealing with mood 
disorders and/or addiction. 

With this perspective, I decided to pursue a PhD in neuroscience to gain skills that would help me 
on my journey to become a translational scientist. In the lab of Andrea Hohmann at Indiana Univer- 
sity, I learned how to conduct behavioral experiments in rodents that serve as useful models for 
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studying the interconnected subjects of neuropathic pain, mood disorders, and opioid addiction. By 
conducting preclinical behavioral work in each of these areas, | have observed an important theme 
that links them: dysregulation in the nervous system is associated with the shift from a behavior that 
is “normal” or “healthy” to a behavior that can be described as “blunted” or “overactive.” Biological 
systems that are typically useful to an organism’s survival (functional pain, stress, and reward process- 
ing) can become problematic for it when the system becomes dysregulated. 


Pain, Stress, and Suffering 


We turn now to defining and differentiating pain, stress, and suffering as understood within the 
humanities and biological sciences. These concepts exist across a spectrum and are the results of 
different stimuli, thus evoking diverse emotional and physical responses. We created a visualization 
(see Fig. 1) to provide an overview of similarities and differences between pain, stress, and suffering 
discussed more in depth in the sections that follow. 


Pain Stress Suffering 
Win 


fF ff 


emotional/physical disturbance to organism's « 


. . prolonged distress that 
unpleasantness associated with homeostasis leads to existential threat of 


Definition actual or potential tissue * environmental/physical an individual's sense of self/ 
damage (temperatures, pressure that evokes a personhood 
chemicals, injury, diseases, response from an organism 
inflammation, etc) 
* motivation for an organism to * motivation for an * motivation to fix source of 
Adaptive change its state to avoid organism to change its suffering (biological, 
Response biological threats to state to avoid biological interpersonal, political, etc) 
homeostasis and survival threats to homeostasis * source of increased personal 
and survival meaning, growth, purpose, etc. 
« Pathological/neuropathic pain ¢ chronic/uncontrollable * Needlessly decreases quality 
Maladaptive causing distress in the stress negatively affects of life 
absence of a threat many body systems and 
Response an individual's well being 
* questionaire (self) * questionaire (self) ¢ Reliance on (unquantified) 
inference through behavior ¢ inference through behavior _ narratives, aesthetics 
Measures (others) (others) 
¢ physiological markers * physiological markers (hormones, 
(nerve conduction, etc) parasympathetic response, etc) 
Fig. 1. Differences Among Pain, Stress, and Suffering 
Pain Defined 


Pain is defined by the International Association for the Study of Pain ([ASP) as “an unpleasant 
sensory and emotional experience associated with, or resembling that associated with, actual or 
potential tissue damage” (Raja et al. 2020, 1977). Recently updated through small changes of lan- 
guage, the previous definition had received critiques for excluding social and cognitive components 
of pain as well as trivializing the extent of peoples’ pain (Williams & Craig, 2016, Wright 2011). 
However, the word “unpleasant” remains the umbrella term for the base characterization of every- 
thing from stubbing one’s toe to breaking a femur. Generally, pain comes from external stimuli that 
inflicts degrees of discomfort on the body or mind but remains separate from self-understanding. 
Elaine Scarry (1985) is well known for pushing beyond strictly medical interpretations of pain by 
applying political and historical lenses in The Body in Pain, where she argues that pain defines reality 
as well as certainty in one’s own experiences. For Scarry, “Physical pain does not simply resist 
language, but actively destroys it” (4). As we are then reliant on metaphorical expressions to articu- 
late our experiences in the face of linguistic destruction, the ambiguity inherent in metaphor also 
requires interpretation from the listener (Lakoff and Johnson 1980). This space of uncertainty and 
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doubt when observing another’s pain (also expressed by Scarry) means that the study of pain can 
often be an imprecise and subjective one. Though an exact method to study pain would prove 
extremely useful, the reliance on the production and interpretation of metaphors to represent one’s 
internal state creates the potential for imprecision. 

David Morris (1991) expands on Scarry’s philosophical framework by discussing cultural aspects 
of pain, particularly in the limited way America medicalizes pain (see also Kleinman 1988). Morris 
also articulates the “Myth of Two Pains,” which commonly divides pain into physical or mental 
types, though this assumption ignores the “interdependence of mind and body” as well as the mul- 
tiple ways in which pain can manifest itself (10). Thinking about pain in the sciences and humanities 
might further this myth of two pains, with science often being drawn to physical pain and the 
humanities using aesthetics to either provoke or alleviate mental pain. Left unresolved, temporary 
pains of the mind and body can lead to suffering, though scholarship on suffering gives particular 
weight to the mind. We see many of the ways that Scarry and Morris describe the significance of 
intense pain more in line with the existential threat tied together with suffering. 

To categorize pain more succinctly, Loeser and Melzack (1999) differentiate pain stemming 
from nociception (the ability to perceive a noxious stimulus), perception of pain, suffering, and pain 
behaviors. Within these types, pain also can have anatomical, physiological, and psychological 
effects (see also Loeser 1980). From a biological perspective, nociception is vital for a species’ sur- 
vival, helping individual organisms to learn about and avoid temperatures, chemicals, and mechani- 
cal damage that would threaten their ability to survive and reproduce. Such cases suggest pain can be 
beneficial in that it acts as a warning sign that something is wrong. 


Adaptive/Maladaptive pain processing 


From an evolutionary perspective, a biological process or behavior can be considered ‘adaptive’ 
ifit helps an organism adjust to its environment, and ‘maladaptive’ if it hampers an organism’s ability 
to change to better fit its environment. Normal nociception is thus adaptive, as normal pain process- 
ing helps an organism learn to avoid potentially damaging temperatures, chemicals, and mechanical 
pressures. Various factors (genetics, diabetes, chemical exposure, injury, etc.) can cause dramatic 
shifts in pain processing that result either in a hypersensitivity to non-noxious stimuli (pain response 
from the touch ofa feather) or pain signals sent in the absence of any obvious tissue damage. This 
type of maladaptive pain processing is often referred to as pathological pain; notoriously difficult to 
treat, it represents a large health burden globally. Pathological pain is highly correlated with depres- 
sion and other mood disorders (Von Korff& Simon 1996, Blackburn-Munro & Blackburn-Munro 
2001). Interestingly, a genetic inability to experience pain can also cause problems, as affected 
individuals are unaware of tissue damage and fail to adjust their behavior, sometimes biting off parts 
of their tongues, suffering burns or bedsores, and often living shorter lives (Cox 2006). 

Maladaptive pain highlights instances where pain is not constructive. In pain disorders, pain can 
be traced to a concrete anatomical region (Kumar & Elavarasi 2016). Explorations of emotional, 
cognitive, and social pain (which occupy the mind rather than the body) often demand different 
interventions than anatomical pain might, which Loeser and Melzack (1999) also note. An indi- 
vidual would provide different descriptions of chest pain—physical discomfort located near one’s 
heart—and heartache—emotional pain resulting from loss, betrayal, or other negative social factors. 
We suggest that ongoing pain in this latter category might fit more neatly in a separate definition of 
psychological suffering rather than biological pain, while emphasizing the interconnectedness of the 
two domains. 


Measuring Pain 


Although challenges exist in identifying and quantifying pain, there are standard methods of 
doing so. Such data allows scholars to examine if and to what extent an intervention has a significant 
effect on a person’s pain. Clinically, this is often quickly done by asking patients to rate their pain 
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level ona scale of 1 to 10 as with Claire’s hospital experiences, but several more detailed question- 
naires can offer clinicians and researchers a more complete picture of the exact nature of one’s pain 
and emotional state (Attal et al. 2018). 

As rodents are notorious for their inability to fill out questionnaires, neuroscientists can only infer 
their internal/aftective states through quantification of behaviors that could relate to pain response 
and affect. In this sense, a scientist operates under a narrative: “Because Group A exhibited less 
preference for sucrose than Group B, A exhibited anhedonia-like behavior.” Preclinical scientists 
rely on various behavioral metrics to assess the level of a rodent’s physical pain and affective state 
(Deuis, Dvorakova, & Vetter 2017). To examine pain, one might use a Von Frey filament, which 
allows a scientist to quantify the amount of force required for a rodent to move its paw out of the way 
in response to a mechanical stimulus; this test is useful for quantifying nociceptive hypersensitivities 
(pathological pain) and evaluate potential treatments for it. Evolutionarily, rodents generally prefer 
to avoid open, brightly lit spaces where they might be vulnerable to predation. In the Light-Dark 
Box test, rodents are allowed to freely explore a brightly lit portion of an arena and a darker, ‘safer’ 
portion of an arena. Ifa treatment or condition increases the amount of time a rodent spends in the 
dark side of the arena, it is typically referred to in literature as producing an ‘anxiety-like’ behavior. 
For a more exhaustive review of how pain and negative affect are studied in preclinical settings, see 
Kremer et al. 2021. 

When attempting to characterize and quantify pain among historical and literary figures, scholars 
face a similar difficulty faced by the rodent behavioral scientist: they, too, are unable to converse 
with their subject of study (the dead and the imaginary). Currently, there are no standard, interpre- 
tive measures for past or imagined pain. In addition to relying on aesthetic descriptions within 
narratives, we might use our own experiences to imagine the described sensation and make judg- 
ments of perceived pain. To our knowledge, no scholarship has investigated potential quantitative 
measures of aesthetic pain that might only be accessible through word descriptions, images, and 
imagination. It might be possible to adapt methods used by Brockington et al. (2021) regarding 
positive (biological) responses to storytelling. 

In order for a painful stimulus to enter our awareness, processing and modulation of nociceptive 
input occurs at various junctions in the body. To illustrate this, we can follow the journey ofa pain 
signal after someone stubs their toe when walking across the room. Receptors that detect mechanical 
pressure in the toe will be strongly activated, sending an electrical pulse down the neurons in which 
they reside. These neurons, housed near the spinal cord, transmit signals to other neurons housed in 
the brainstem and the thalamus, a brain region that acts as a relay station and integration center. 
From the brainstem and thalamus, signals are quickly sent to other brain regions that allow the toe- 
stubber to alter their behavior and respond immediately to the injury. As highlighted in Legrain et 
al (2009), Functional Magnetic Resonance Imaging (fMRI) studies indicate that the anterior cingu- 
late cortex (ACC) is a brain region that plays a role in orienting one’s attention to a nociceptive 
signal, potentially due to its strong connections with premotor areas (involved in motor planning) 
and the prefrontal cortex (involved in executive function/decision making). The thalamus also sends 
nociceptive information to the somatosensory cortex, which encodes the origin and timing of the 
stimulus, and the insula, which plays a role in encoding the emotional aspects of pain. As reviewed in 
Bushnell et al (2012), nociceptive signals from the periphery are also relayed through the amygdala 
and the periaqueductal grey, which are also highly involved in emotional processing, fear, and 
fight-or-flight responses. Studies of chronic pain patients document functional and/or neurochemi- 
cal dysregulation in most of the above-mentioned brain regions and targeting aberrant activity in 
the brain regions serves as an attractive strategy to alleviate pain (Narita et al. 2006, Maarrawi et al. 
2007, Wood et al. 2007). 

While our own research interests center on limiting maladaptive pain, we acknowledge not every 
scholar and artist views their work as having an activist or even moral bent. Indeed, some artists are 
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actively aggressive in their work and seek to cause cognitive pain and discomfort, such as in Italian 
futurism and avant-garde performances around metaphorical and physical ruptures (Perloff 2003). 
For these artists, transforming art or literature into a type of violence might be comparable with 
adaptive pain: just as physical discomfort can result in evolutionary benefits, emotional or cognitive 
discomfort can provide individual or societal benefits. Faced with an art exhibit or collage that 
performs or provokes pain (for example, witnessing past traumas through music, see Cizmic 2012), 
a viewer might be provoked into reflecting on and working through the causes and effects of 
psychological pain. Creators might also promote cognitive rupture for an audience to experience 
pain simply for pain’s sake, considering the act of experiencing pain itself a worthy aesthetic expe- 
rience. In “The Futurist Manifesto of Lust,” Valentine de Saint-Point writes, “It is the painful joy of 
wounded flesh, the joyous pain of a flowering” (1917). Such paradoxical images of pain do not 
necessarily imply masochism, but rather emphasize an ongoing, polemic dialectic between the 
aesthetics of joy and pain. 


Stress and resilience 


Much like pain, people often view stress in a negative light given its contribution to many adverse 
mental and physical health outcomes. The concept of stress as we understand it today was first 
introduced into biomedical contexts by endocrinologist Hans Sleye, who suggested that stress was a 
“nonspecific response of the body to any demand” (Tan & Yip 2018). Selye’s work documented how 
various “stressors” in an organism’s environment (food deprivation, exposure to extreme cold, sleep 
deprivation, harmful chemicals) caused similar deleterious effects on several organs in rats. Biologi- 
cally, there are several well-defined biological pathways that experience increased activity in re- 
sponse to an environmental stressor. The hypothalamic-pituitary-adrenal (HPA) axis consists of the 
adrenal glands and two brain regions called the hypothalamus and pituitary gland, which release 
“messenger” hormones into the bloodstream in response to a stressor. The adrenal glands subse- 
quently release cortisol into the bloodstream, which arouses the body and initiates a ‘fight or flight’ 
response. We may not consciously think about these biological processes when we watch a horror 
film or realize we forgot about a work assignment, but we are all familiar with the heightened mental 
awareness and anxiety that accompany stress. 

Although most people might not describe their stress as painful, stress can be an unpleasant sensa- 
tion reaching back to the core medical definition of pain. Koolhaas et al. (2011) suggests that stress 
“should be restricted to conditions where an environmental demand exceeds the natural regulatory 
capacity of an organism, in particular situations that include unpredictability and uncontrollability” 
(1291). Without safety nets for regulating stress, it can become overwhelming. Ongoing stress has a 
direct link with suffering (Quick & Henderson 2016). Similar then to unaddressed pain, continual 
high stress levels can lead to suffering. However, not all interpretations of stress directly lead to 
suffering or even negative outcomes. Healthy levels of stress can also motivate us to complete impor- 
tant tasks or remove ourselves from a dangerous situation (McGonigal 2016). From an evolutionary 
perspective, an organism that does not experience any stress in the presence of a predator would be 
much less likely to survive and reproduce. When we are engrossed in a novel, TV series, or even 
artwork, we might feel stress (or relief) from the intensity of the story or image presented. This 
reaction, rooted in an evolutionarily advantageous biological response and regulated by various 
systems in the body and brain, emphasizes the interconnectedness between the brain, body, and 
mind in aesthetic experience. 

Although stress is not necessarily a cause of disease per se, chronic or uncontrollable stress can 
exacerbate or increase the risk for the development of a wide variety of physical illnesses and mental 
disorders such as depression, anxiety, or post-traumatic stress disorder (Salleh 2008). When exam- 
ining stress’s role in the development of mental disorders, it is helpful to consider the concepts of 
resilience and vulnerability to stressors. For example, while many war veterans are exposed to trau- 
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matic events during their service, only subpopulations will develop mental disorders (Fogle 2020). 
Currently, scholars from many disciplines are engaged in understanding the biological, psychologi- 
cal, and social factors that play into stress resilience. Such resilience and stress management skills can 
prevent momentary pains of stress from turning into sources of ongoing suffering. 

Though in this article we articulate pain, stress, and suffering as separate concepts, we also recog- 
nize the convergence of their physical manifestations in the body. For example, by measuring 
electrodermal activity, a Galvanic Skin Response (i.e. increase in sweating) can indicate pain level 
(Agajari 2021) or emotional/mental state in a patient (Markiewicz 2021). 

The hypothalamus acts as a crucial link between the brain and the body, as it receives neural 
information from many brain regions that process pain/stress and regulates the release of hormones 
(cortisol, oxytocin, among others) into the bloodstream. Interestingly, studies have shown dysregulation 
of cortisol (too much or too little) among chronic pain patients, highlighting the need for interven- 
tions that are helpful in both domains (Hannibal 2014). Recently, researchers found that storytelling 
decreased cortisol levels and pain scores in hospitalized children, while simultaneously increasing 
oxytocin levels and positive emotions (Brockington 2021). Interestingly, storytelling as an interven- 
tion was superior to a control social intervention (solving riddles), illustrating the unique and poten- 
tially useful therapeutic potential of storytelling in clinical settings. The high degree of neuroana- 
tomical and functional connection between the brain regions that process stress and pain highlights 
the (literally) interwoven relationship between how we think and feel, how we process pain and stress, 
and how dysregulation in their functionality might combine to produce what we call suffering. 

Acutely, stress can suppress pain, while chronic stress can amplify ongoing pain. Additionally, 
chronic pain can introduce stress into an individual’s life, creating a vicious cycle that can culminate 
in an individual’s extended suffering. Alternatively, artistic expression and storytelling, incorporated 
in some models of therapy, can reduce pain as well as stress through increased cognitive and affective 
processing, limiting the more negative effects of suffering. The physiological, neuroanatomical, and 
experiential interplay between pain, stress and suffering highlights the potential for holistic inter- 
ventions based in aesthetic experience (i.e. art, music, narrative therapy) to compliment strictly 
biological approaches in medicine. Understanding the biological mechanisms of storytelling could 
potentially guide its incorporation into clinical settings and bolster its legitimacy as a therapeutic 
strategy for treating pain and stress. 


Suffering 

As mentioned in our previous sections, both stress and pain can lead to suffering, which we 
consider a longer and more intense negative experience, as well as one more concerned with the 
perception and preservation of the self. The transition from pain and stress to suffering fundamen- 
tally involves medical perspectives with holistic conceptions of who we are. It is especially here that 
the collaboration of neuroscience, medicine, and the humanities is needed. 

Instead of concentrating on defining suffering itself, much of the existing literature on suffering 
examines the effects and possible meaning of suffering and its effects of self-understanding or self- 
actualization. Philosophy has long pondered the transformative potential of suffering and its 
contrastingly nihilistic nature (see for example Nietzsche 1886, Bataille 1989, Schopenhauer 2010), 
as well as ways to transcend everyday suffering, exemplified in Buddhist psychology and practices 
(Burton 2016). Frank (2001) reflects on whether research on suffering is possible in that “suffering 
resists definition because it is the reality of what is not” (355), or rather, it is less observable and 
measurable than pain in most instances. 

Nevertheless, scholars in the humanities and sciences have negotiated several characteristics of 
suffering. Reich (1989) contrasts pain with suffering, defining pain as “acute or chronic physical, 
mental, or emotional distress associated with...[an] unpleasant stimulus characterized by discom- 
fort, which the mind perceives as an injury or threat of injury” to the body (85). Suffering, he goes 
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on to suggest, “is of a different order” than pain— it is existential anguish over such an injury or threat 
of injury. Again, the adjective “unpleasant” covers a wide variety and intensity of suffering, but its 
existential nature signifies a greater connection to the self. Suffering can, of course, extend beyond 
the typical anatomical nature of pain. Medically speaking, it is the “perception of serious threat or 
damage to the self” (Chapman & Gavrin 1999, 2233). 

Similarly, Kanov (2020) proposes that suffering is “the severe or protracted distress people expe- 
rience when an instance of pain or injury (emotional, physical, or otherwise) disrupts one’s basic 
personhood” (86). There is then a focus on preservation of the self. In this sense, suffering can be 
viewed as more of an issue of sustained and uncontrollable harm to an individual, a mind, ora soul 
rather than the immediate consequences of tissue damage. We see an element of chaos included in 
these definitions of suffering where instability negatively impacts the self, though such chaos may 
also lead to reflection on and transformation of the self. 

For Wallace (2014), suffering includes “undergoing—or actively, allowing oneself to be affected 
by—some outside force, most often pain or distress” (19). In her work on the literary ethics of 
suffering, Wallace focuses on “women borne” and the distinct ways the literary female suffers. She 
interprets suffering as “both the action [cause] and the pain or distress itself” and thus part of the 
passivity that “typically take the female gender association in the Western binary that associates 
activity with maleness” (20). In this way, suffering may be gendered and interpreted as something 
one must endure rather than actively seek alleviation. Though we do not go into detail here, we 
recognize that medicine historically and at present does not value all articulations of pain equally, 
often discounting or ignoring women’s pain, especially Black women, which leads to unnecessary 
ongoing suffering and preventable deaths (Hoffman & Tarzian 2001, Chemaly 2018, Cooper Owens 
2017, Cleghorn 2021). Even in preclinical research, historically the majority of studies only exam- 
ine male subjects (Zakiniaeiz et al 2016), leading to a 2016 mandate by the National Institutes of 
Health for researchers to address sex as a biological variable in preclinical studies. Seemingly “objec- 
tive” measurements of pain still require interpretation by healthcare providers (as well as research- 
ers) which thus contribute to structural inequities in healthcare and unresolved suffering (Bailey et al 
2021). The misjudgment and disbelief of the pain and suffering experienced by marginalized groups, 
including Black and Indigenous populations, people of color, trans individuals, those with disabili- 
ties, and women, should be rectified. Narrativizing pain is a way to share one’s lived experiences. 
Though there is a spectrum of precision in measuring pain and suffering, the stories we tell ourselves 
and others affect the way we experience pain, stress, and suffering as well as how others respond. 


Measures and Meaning of Suffering 


Suffering often requires lengthy descriptions regarding negative effects on the self (Charmaz 
1999). Thus, suffering often comes in the form of lengthier narratives instead of brief descriptions or 
quantitative measures. If pain can be traced to the receptors and cell activity in the brain, suffering 
might be metaphorically tied to the soul. Limited attempts have been made to measure descriptions 
of pain through words rather than numbers, such as the McGill Pain Questionnaire (Melzack 1975), 
though such questionnaires are not standard. Narratives offer a way to both explain and attempt to 
understand one’s suffering, and usually the more complex or poignant the suffering, the more de- 
scription it requires. A 10 ona scale from 1-10 simply cannot cover all the distress one is experienc- 
ing, though attempts have been made to create a self-reported suffering scale (Vander Weele 2019). 
Suffering may be more accurately depicted through phases that reflect a transformative arc. Reich 
(1989) offers three stages of suffering: 1) mute suffering, 2) expressive suffering (narrativizing and 
interpreting one’s experiences), and 3) finding a new identity in one’s suffering (see also Sélle 1984). 
For us, the suffering narrative can be termed asa tale of strong, negative emotions, often interwoven 
with attempts of meaning making. 

Suffering should be taken seriously in and outside of medicine despite the difficulty in articulating 
its complexity and effects on physical, emotional, mental, and social well-being (Chapman & Gavrin 
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1993, 1999). Healthcare systems have traditionally relied on biological approaches to understand 
pain; however, suffering can extend between these processes and demands different modes of mea- 
surement and invention. Unlike pain/nociception, which has evolutionary benefits, suffering can 
often stem from issues in biological and social systems. For example, normal pain processing systems 
gone awry can send ‘pain’ signals to the brain even in the absence of anything harmful to an indi- 
vidual, resulting in excruciating discomfort that can last for decades. Clinically, patients afflicted 
with chronic pain conditions often experience comorbid depression and anxiety (IsHak 2018, Von 
Korff 2018). Similarly, dysfunction in family, community, and national relationships can create 
sustained suffering for members experiencing abuse, oppression, and war. The interpretation of 
suffering can be much less measurable than the warning signs attached to pain. 

There are different schools of thought and religious traditions around suffering, as well as its 
relationship with individual agency. Ancient schools of thought like Stoicism and modern sciences 
like Cognitive Behavioral Therapy share the aim of overcoming suffering. In these approaches, 
pain need not be a negative aspect of life, but the reactive temptation to dwell with pain can lead to 
suffering. Many modern approaches and motivational speakers may link choice and suffering, fo- 
cusing on such adages like pain is mandatory, suffering is optional. 

Though most would agree suffering is disruptive and negative, many people find meaning in 
their own suffering as well as that of others, especially in religious and philosophical contexts. A 
simple search on suffering at the public library in Bloomington brings up self-help genre titles such 
as Inside the miracle: enduring suffering, approaching wholeness (Nepo 2015), Silence and beauty: hidden 
faith born of suffering (Fujimura 2016), and Tears to triumph: the spiritual journey from suffering to 
enlightenment (Williamson 2017). Suffering, in these contexts, acts as an impetus for personal growth 
and also offers connection to others. In this sense, suffering can potentially lead an individual froma 
state of spiritual poverty (Helminiak 2020) toa state of deep meaning, purpose, and connection with 
the divine. Stress, too, can be motivating in accomplishing tasks, removing oneself from misfortune 
when possible, and providing evolutionary benefits (Hoffman & Hercus 2000). Increased empathy 
for those who suffer can be a prosocial behavior that drives connection and understanding as well as 
decrease loneliness (Klimecki 2019, Konrath & Grynberg 2016, Batson & Ahmad 2009), with 
empathetic behavior connected with observing pain even being found in mice (Langford 2006). 
However, empathy may also lead to increased polarization (Breithaupt 2017) and competitive 
victimhood (Noor et al 2012). By pointing out some of the perceived benefits to suffering, we do not 
mean to trivialize the very real physical and mental health issues associated with suffering. Negative 
outcomes including violence against self or others should be minimized. 

Bruneau, Pluta, and Saxe (2012) found that reading stories about either physical pain or emotional 
suffering reflected similar patterns for both in neuroimaging. However, stories about physical pain 
indicated higher responses in the Shared Pain Network whereas those about emotional experiences 
demonstrated higher responses to areas corresponding with the Theory of Mind Network, or the 
ability to understand how another person is thinking and feeling (229). Even imagining another’s 
pain demonstrates the important differences in neuroprocessing between types of pain. 

Many have documented the adaptive effects of suffering, using more aesthetic methods such as art 
and literature as measures (Musso & Enz 2016). De Vries (2012) examines the ways musicians and 
artists allow and even engage with suffering in their works rather than avoiding it altogether. 
Similar to our comments on aesthetic works that evoke pain, creators may actively work to perform 
and reflect suffering through poetry, art, music, narratives, and film among other mediums. These 
works are not always meant as a cathartic process that leads to a harmonious end like Aristotle may 
have envisioned of drama, and indeed characterizing suffering as an inevitable resolution does a 
disservice to the parts of suffering that cannot be articulated (Edgar 2007). Indeed, attempts to 
represent the unrepresentable might further suffering instead of providing a space of healing. Adorno’s 
oft quoted but seldom contextualized statement on the impossibility of poetry after the Holocaust 
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(1949) highlights the unrepresentable nature of this particular atrocity and the incompatibility of 
mimesis (replication) with real, felt suffering. This, of course, has not stopped people from identify- 
ing and representing various forms of trauma. 

At the risk of overstepping such boundaries ourselves, we suggest a network of dynamic, interact- 
ing factors that can begin as stress or pain and develop into suffering, including physical, mental, and 
emotional anguish in addition to the limited existing quantitative measures. Diagnosing the kind of 
stressor(s) that lead to suffering can be helpful in then identifying appropriate and effective resolu- 
tions. We offer a starting point for categorizing suffering in Table 1. This categorization is not a 
comprehensive list of stressors and types of suffering, but a beginning for future work to explore. 


Type of Stressor/Suffering Description 
Loneli social isolation and rejection, grief and emotional 
ical devastation after losing a loved one 
: lacking basic freedoms, such as those living under 
Oppression 
dictatorships 
Physical torture intentional infliction of severe pain 


watching someone in pain or dying (varying degrees of 


Helpless Observation 
: intimacy/affectedness), empathy acts as impetus 


Unfulfilled expectations desperately desire for something without obtaining it 
(house, job, partner, baby, etc) 

Poverty hunger, lack of housing and important physical resources, 
dehumanization 


physical, sexual, religious, familial, systemic, racial 


physical or physiological injury 


Trauma and abuse 


Guilt regret for a past wrong 
Addiction failure to control a repeated behavior despite ongoing 
negative consequences (Goodman 1990) 
Disease illness adversely affecting the body or mind 
Unaddressed pain chronic or ongoing pain that is not properly managed 
going p properly g 
Natural Disasters adverse effects from extreme weather that limits 


resources, destruction of homes and community 


War armed conflict between two or more groups that results 
in death, wounds, and loss 


Spiritual Poverty Lack of meaning or belonging, experiences of existential 
angst (often paired with “an abundance of material 
goods,” see Helminiak 2020) 


Table 1. Types of stressors and suffering along with their descriptions. 
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In addition to systematizing types of suffering as a tool for identifying effective interventions in 
medical settings, a potential formal questionnaire might help individuals, medical providers, thera- 
pists, advocates, and scholars differentiate between individuals experiencing stress and those under- 
going suffering. Questions might measure frequency and intensity of the stressors mentioned above. 
We also suggest open-ended questions where individuals can narrativize their experiences in order 
to provide important effects that might be missed through quantitative measures, a common prac- 
tice in standard therapy. When attempting to characterize suffering in historical, literary, or artistic 
contexts, a quantitative framework might also be helpful and innovative for narrative theory. Using 
a list of official stressors as a guide, scholars could determine frequency and interpret intensity of each 
stressor within a narrative or work. Frequency and intensity measures might be aided by AI technol- 
ogy such as sentiment analysis and textual analysis, common tools already used in digital humanities. 


Implications and Conclusion 


In exploring the relationships among pain, stress, and suffering, this article was also an exercise in 
bridging disciplines to discuss concepts that have universal importance. We were surprised by how 
difficult it was to not only finda common vocabulary but also agree ona direction for this article and 
find commonalities within the larger study of pain. While one of our goals was to document methods 
of studying pain, stress, and suffering, we also have come away from the cowriting experience with 
increased commitment to collaborative work on the matter. Examining how mind and brain work 
together in processing and articulating moments of physical agony, heightened anxiety, and un- 
bridled grief influences the narratives we create about these experiences as well. Traditionally, the 
sciences attempt to explain and describe natural laws that govern our world, while the humanities 
can offer interpretations and deeper meaning of these laws and the lived experiences of those in the 
world. However, conversations between these disciplines, common in narrative medicine and bio- 
ethics, create innovative solutions for problems related to negative health outcomes and offer more 
complete narratives. 

Positive interventions in pain, stress, and suffering often rely on interdisciplinary research and 
play a critical role in understanding pain and suffering and reducing unnecessary kinds of both. 
Interdisciplinary is usually talked about between two very similar fields, such as projects incorporating 
molecular biology and biochemistry or history and literature. However, we advocate for projects 
that utilize knowledge from diverse fields that further creative insights in our understanding of how 
the arts and sciences come together in unique ways. 

Creating spaces where scholars from vastly different fields can come together to discuss individual 
projects as well as collaborate with others is an important step in combining and thus furthering 
scientific and humanistic knowledge. Epstein (2014) reviews some of the challenges and advantages 
in making interdisciplinary work a collaborative process such as funding, attitude, communication, 
time, institutional climate, and proximity to other scholars (see also Nancarrow 2013, Bronstein 
2003, Brewer & Lévgren 1999). However, such work is worthwhile in the ways collaborative 
efforts expand and deepen knowledge on a subject, even helping scholars overcome common biases 
within their own disciplines (Repko & Szostak 2020). 

The way that we perceive and connect pain, stress, and suffering is important because it affects 
medical treatments and other interventions on both small and large scales. While accepting the 
presence of pain and suffering in our world and even embracing adaptive aspects of both, we care 
about definitions and relationships because we care about trying to minimize harmful suffering. 
Investigating some of the biological mechanisms that mediate aesthetic experiences has profound 
potential to guide aesthetics for therapeutic benefit in contexts of both physical and emotional pain. 
More effective collaboration and improved common language between basic scientists who study 
pain in the brain and humanities scholars who study depictions of suffering in relation to the mind 
and conceptions of self would facilitate more effective and creative solutions to alleviate problems 
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relating to pain and suffering. Understanding the extent of pain and suffering can encourage prosocial, 
empathetic, and effective interventions that range from medicinal treatments to art, music, or 
storytelling therapy. 

Individual differences in the ability to cope with pain, stress, and suffering also reveal the need for 
a variety of solutions in decreasing unwanted pain, stress, and suffering. One person’s rating of a 4 on 
a 1-10 pain scale might physiologically be the same as another person’s 7, and each of these individu- 
als may desire different interventions when it comes to possible medication or therapies. An array of 
factors can affect the way one experiences pain, including demographic variables, genetic factors, 
and psychosocial processes (Filligham 2017). External and internal influences on the way one expe- 
riences pain, stress, or suffering provide another example of the challenges in creating objective 
measures of these concepts. We emphasize the importance of ongoing research to effectively pair 
aesthetics-based interventions with the appropriate clinical indication. Understanding individual 
differences both in pain/stress processing as well as responsiveness to different aesthetic experiences 
will be important for determining which therapies are applicable to which individuals for which 
difficulties they are facing. 

Pain and stress have the potential to negatively impact our lives, while suffering can affect the 
perception of one’s very sense of self: The aesthetics of pain, stress, and suffering influence how we 
interpret each of them in addition to the quantitative or narrative descriptors those in pain may 
provide. Further research may look to fill in gaps between cognitive neuroscience and aesthetics 
regarding pain and suffering, such as examining efficacy of new questionnaires, expanding and 
analyzing categories of stress and suffering, and continued experimental approaches to interpreta- 
tions of pain. 
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Abstract: Supernormal stimuli were first defined in ethology as exaggerated forms of natural or 
artificial stimuli able to elicit amplified responses in animals. After the characterisation of supernor- 
mal stimuli in ethology, many scholars have explored the idea of supernormality also with reference 
to human cultures, resulting in the assignment of this attribute to several categories of stimuli, from 
pornography to fast-food. In the last decades, also art has been proposed to act as a supernormal 
stimulus for human beings. After reviewing the major contributions in this field, here we explore this 
hypothesis by highlighting experimental evidence from studies in neuroaesthetics and art history 
cases that corroborate this view in an attempt to show the potential that this framework can bring to 
the aesthetic debate. 


Keywords: aesthetics; art; supernormal stimulus; neuroaesthetics. 


Definition and Origins of the Supernormal Stimulus 


he term supernormal sign stimulus — also named supernormal stimulus — refers to a stimulus 

capable of producing reactions and behavioural patterns that are amplified compared to the 
norm (Tinbergen, 1948, 1951; Immelmann and Beer, 1989; Barnett, 2021). The phenomena re- 
lated to supernormal stimuli were first systematically described as part of the studies on the instinc- 
tive behaviour of animals, conducted by Konrad Lorenz and Nikolaas Tinbergen between the 
1930s and 1950s. In those years, Lorenz advanced the imprinting theory, according to which 
instinctive behaviours are activated by innate trigger mechanisms that are provided by specific 
stimuli, defined as “sign-stimuli” or “releasers” (Lorenz, 1950). Subsequently, Tinbergen observed 
that these innate responses were amplified when the sign-stimulus had physical characteristics exag- 
gerated as compared to the norm, suchas oversize or greater figure-background contrast (Tinbergen, 
1948, 1951). For example, while observing the hatching behaviour of female cuckoos, Tinbergen 
discovered that they stealthily lay their large eggs in the nests of birds of other species, whose eggs are 
typically smaller. The “host mothers”, unaware of the deception, are attracted by the size of the 
cuckoo’s eggs and respond by sitting on the eggs for longer than their own (Tinbergen, 1951). 
Similar behavioural responses were also observed in the presence of artificial stimuli. For example, 
Koehler and Zagarus (1937), years before Tinbergen, observed that ringed plovers (Charadrius 
hiaticula) preferred to hatch artificial eggs with characteristics amplified by greater figure-back- 
ground contrast than their own. Similar phenomena in animal behaviour have been observed also in 
other species such as insects and non-human primates, both when faced with natural and artificial 
stimuli (Tinbergen, 1948, 1951; Tinbergen and Perdeck, 1950; Magnus, 1958; Gray et al., 1980; 
Bielert and Anderson, 1985; Tanaka et al., 2011). It has been hypothesised that there is an evolution- 
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ary mechanism of “asymmetric selection” at the basis of the effects that a supernormal stimulus can exert 
in nature (Staddon, 1975), according to which it is the absence of selective pressure against amplified 
behavioural responses that allows for their manifestation. Similarly, the “peak shift” effect has also been 
hypothesised to rely upon this type of evolutionary mechanism. Peak shift is a behavioural phenom- 
enon — observed in learning processes and related to that of supernormality — that based on previous 
experience causes a shift in preference towards items with specific characteristics (Hanson, 1959). 

As of today, the original conceptualisation elaborated by Tinbergen has been mostly abandoned 
by the field of ethology (De Block and Du Laing, 2010). This happened for different reasons, among 
which is the fact that the original conceptual framework was drawing from the contemporary 
theoretical assumptions, mainly based on an innatist point of view (De Block and Du Laing, 2010), 
which are mostly outdated nowadays. Nevertheless, the concept of supernormal stimulus has ex- 
panded beyond the animal kingdom and it has subsequently been linked to many phenomena of 
human culture, for instance the spread of television and video games (Barrett, 2010), fast-food 
(Saad, 2011) and junk-food consumption (Witherly, 2007), control over bodily appearance 
(Pazhoohi et al., 2020a, 2020b; Morris et al., 2013), pornography consumption (Barrett, 2010), 
advertising and marketing (Hendlin, 2019), the Internet (Ward, 2013), terroristic attacks (Geher, 
2015), and religion (Boyer, 2007). On top of this, greater attention has been devoted to aesthetic 
experience, by discussing the hypothesis that visual arts (Campbell, 1959; Coss, 1968; Behrens and 
Whitson, 1976; Ramachandran and Hirstein, 1999; Costa and Corazza, 2006; De Block and Du 
Laing, 2010; Grinde and Husselman, 2022), music (Knobloch, 1995/2000), and literature (Nettle, 
2005), do indeed act as supernormal stimuli for humans. 


Supernormal Stimuli in Human Culture 


Based on the original ethological concept, there are two main levels to be taken into account when 
talking about supernormal stimuli: the one of the object per se, which carries with it amplified 
physical and formal characteristics — such as higher figure-background contrast, greater dimen- 
sions, reduction to essential characteristics, etc. —, and the one of the subject that while interacting 
with it reacts with amplified behavioural responses — by showing higher frequency of a behaviour 
and overt preference for that specific item (Gagliardi et al., 2022). 

In overcoming the initial specificity of the ethological definition ofa supernormal stimulus, there 
have been several attempts to apply this concept to phenomena of human culture, many of which 
belong to the field of evolutionary psychology. Here we will review the major attempts that have 
been proposed in the last two decades, by going through those studies that borrowed the concept of 
supernormal stimulus, or an adapted definition of it, when dealing with phenomena of human 
culture. It is important to notice that some of these theories have an inherent speculative nature, and 
thus various alternative and novel theories could be proposed. 

American psychologist Deirdre Barrett in her popular book Supernormal Stimuli. How primal 
urges overran their evolutionary purpose (2010) was among the first to dig into various phenomena of 
today’s consumer society, such as pornography, cosmetics, fashion, advertising, fast-food, manga, 
anime, and the media and entertainment industry. Barrett suggests that we can conceive all these 
products of human culture as supernormal stimuli because of the accentuation of certain peculiar 
characteristics of these stimuli, whose impact reflects on activating and attracting the consumer to a 
greater extent. Nevertheless she speculates that these categories of phenomena can become mal- 
adaptive, as they activate the reward systems to a level not found in natural stimuli (see also Goodwin 
etal., 2015, for a similar perspective). In this regard, Barrett considers how these stimuli could have 
a negative effect on our health, being the driving force behind certain problems peculiar to our 
societies, such as obesity, depression, anxiety, and various forms of addiction. Interestingly — she 
sustains —, this occurs regardless of the fact that human beings — unlike animals — once aware of the 
supernormal character of a stimulus, would be able to apply logic and self-control to manage and 
overcome responses of an instinctual nature (Barrett, 2010). 
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From an evolutionary perspective, it is still unclear why the exaggerated forms of certain stimuli 
are more attractive, although some mechanisms have been proposed to explain the roots at the basis 
of such a phenomenon (see Staddon, 1975). One speculative hypothesis has been put forward by 
Barrett herself to explain how these stimuli can exert a maladaptive function nowadays, by suggest- 
ing that the evolutionary root of our preference for these types of stimuli is to be found in the fact that 
our reaction to them evolved in individuals living 10,000 years ago, in environments consistently 
different than ours and characterised by a paucity of these types of stimuli. On the contrary — she 
continues — the high number of exaggerated artificial alternatives of supernormal stimuli present 
nowadays challenges our ability to manage and deal with them (Barrett, 2010). 

A similar analysis has been proposed by psychologist Steven Pinker by using the metaphor of the 
“cheesecake for the mind”. Specifically, the author refers to the many cultural stimuli that attract us 
and succeed in doing so because they are capable of activating brain circuits that have evolved to 
make us experience sensations of pleasure and that originally had an adaptive purpose (Pinker, 
1997). In this sense, stimuli such as drugs — which Pinker includes in the classification of supernormal 
stimuli together with others, among which is literature — would be artificial stimuli that contain 
«megadoses of stimuli that cause us pleasure» (Pinker, 1997, p. 524). Pinker’s position in this regard 
appears quite speculative, and other authors have come forward in highlighting its limitations and 
incompleteness. One of these is Carroll (1998), who has criticised Pinker’s views for not showing 
how different supernormal stimuli can lead to maladaptive behaviour while others can be highly 
adaptive. For example, the author points out that recreational drugs have been proven to have 
potential negative effects if taken during the developmental period, while phenomena like art and 
music — that are also considered supernormal stimuli — are instead to be deemed positive from 
different points of view, over the same years. 

According to some analyses, many entertainment devices that have to do with storytelling, such as 
television, films, comics, novels, and TV series, are to be considered supernormal stimuli for the 
audience. For example, different studies exploring the evolution of narrative fiction or the viral 
video-information phenomena (see for example, Nettle, 2005; Barrett, 2010; Astolfi, 2012; Arielli 
and Bottazzini, 2018; van Peer, 2018; Burch and Johnsen, 2020; Burch and Widman, 2021; Dubourg 
and Baumard, 2022) have focused on identifying narrative and image manipulation devices that 
have the effect of activating and attracting the viewer above normal. Think of the narrative devices 
used in fiction to amplify the reader’s engagement through the exaggeration of events in which the 
main character is involved, who often has to go through a series of out-of-the-ordinary trials to gain 
prestige or achieve particular goals. In some other cases, it is the text that is exaggerated in fiction, for 
instance through the employment of rhetorical figures. Not to forget also exaggerated forms of 
animated characters and comic strips, which amplify normal human forms to make them more 
attractive (Burch and Johnsen, 2020; Burch and Widman, 2021); or the auditory stimuli of many 
film genres, such as horror or action films, in which we hear terrifying sounds that are amplified 
(Gallese and Guerra, 2019). Finally, probably among the most evolved forms of entertainment, are 
immersive video games and virtual reality, which reconstruct an entire environment by amplifying 
its forms and the beings that inhabit it. 

Similar devices are also to be found in advertising and communication, where stimuli are thor- 
oughly designed to be increasingly refined and attractive. In advertising, some scholars have de- 
tected the use of supernormal signs and symbols for deceptive and non-evolutionary purposes. 
Products placed on the contemporary market are designed, manipulated, and commercialised to 
attract and produce awe in the user in order to maximise sales and profits, often with little regard to 
social and physical consequences (Hendlin, 2019). 

As part of the study of media communication, even terrorist attacks have been proposed as super- 
normal stimuli for humans (Geher, 2015). The strong negative reaction of human beings to stimuli 
such as blood and the gory scenes associated with these types of attacks — whose media resonance is 
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practically inevitable — causes these events to be deemed, according to some interpretations, a form 
of supernormal stimulus precisely because of their reaching thousands of people, and thus being 
capable of producing mass results of negative psychological nature (e.g. Bildhauer, 2013). Whereas 
in ancestral societies it would have been impossible to reach such large numbers of people, in con- 
temporary society — through media and the dissemination of images — terrorist attacks easily act on 
a large scale. In this specific case, the association with the concept of supernormal stimulus heavily 
relies on the amplified amount of people reached worldwide witha single event. 

As previously suggested, the likelihood of many of our society’s supernormal stimuli becoming 
harmful or maladaptive is high. However, this is not always the case according to other studies. The 
global-level telecommunications network represented by the Internet has been identified by Adrian 
F, Ward (2013) and others (see e.g. Firth et al., 2019) as a form of supernormal stimulus for humans. 
In his work, Ward starts from the consideration that «the Internet may act as a “supernormal stimu- 
lus”, hijacking preexisting cognitive tendencies and creating novel outcomes» and he refers to the 
notion that «supernormal stimuli meet or exceed long-enforced selection criteria, but are generally 
foreign to the environments in which these criteria developed; asa result, these new stimuli often 
elicit greater responses than any naturally occurring stimuli» (Ward, 2013, p. 341). By identifying 
this conceptual framework, Ward argues that the Internet may act as a supernormal stimulus in 
many domains, especially that of memory — transactive memory in particular —as it can increase the 
information storage potential compared to the limited biological storage capacity while decreasing 
the related cognitive effort, hence exerting a pervasive influence by going so far as to hijack an 
individual’s pre-existing cognitive tendencies with both positive and negative consequences. As 
important an issue as this is to be tackled in the present times, the arguments advanced by Ward in 
favour of including the Internet in the category of supernormal stimuli seem to deviate significantly 
from the original definition of the term, which relies more on the characteristics of the stimulus per 
se and the beholder’s specific response to them. 

The reported phenomena have been identified as forms of supernormal stimulus or linked to it in 
the context of human culture, and the related disadvantages for the individuals and the society have 
been often outlined in these studies. However, as one can notice, in many cases the types of stimuli 
included in the definition of supernormal stimulus do not strictly conform to the original definition 
and this might result in speaking of supernormality when dealing with stimuli that just evoke a 
stronger (as opposed to normal) response — be it “intense”, “frequent”, “long” or even “reaching many 
people” (as in the case of terrorism) or “change the normal cognitive potential” (as in the case of the 
Internet). Moreover, as we have seen before, not all supernormal stimuli in our societies are harmful; 
some of them, on the contrary, are said to increase the “fitness” of the recipient (De Block and Du 
Laing, 2010) -ice., the ability to survive and reproduce. De Block and Du Laing’s reflections open 
up the discussion of the central topic of our research, namely art as a supernormal stimulus. 

In the following sections, we will look at how art and aesthetic experience — just like other cultural 
products — are indeed characterised by “modified” and “exaggerated” forms and are capable of 
activating spectators in a complex way, actually providing benefits. 


Art as a Supernormal Stimulus 


Aesthetic experience encompasses the set of bodily and mental activities that human beings en- 
gage in when confronted with an artwork (Leder et al., 2004; Jacobsen, 2010; Leder and Nadal, 
2014; Beudt and Jacobsen, 2015; Chiarella et al., 2022). Such activities are to be considered transfor- 
mative for human beings, as the viewer is perceptually and cognitively involved in a non-straight- 
forward level of interpretation (e.g., Menninghaus et al., 2020). In this sense, an artwork, be it an 
object, a ballet, a film, a musical piece, or a literary work, can be defined as such when it loses its 
temporary qualities in order to acquire more universal and abstract meanings and levels of interpre- 
tation: L.e., a painting of a human body becomes the highest synthesis of its aesthetic representation 
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and that of a black silhouette can create psychological interference by altering the emotional and 
perceptive state of the viewer, while the love relationship between two characters can become 
representative of ideal love (Zeki, 2001, 2008). 

Going back to the speculations made by researchers and scholars around the concept of supernor- 
mal stimulus in relation to art, in this section we will review some of the major attempts. Indeed, as 
Ellen Dissanayake suggests (2009), artists perform actions and display behaviours similar to those 
described by ethologists in the ritualised behaviour of animals: they simplify, formalise, repeat, 
exaggerate, and elaborate ordinary materials, bodies, environments, tones, body movements, se- 
mantics and syntax, motifs, ideas. This way, artists «make these elements special» (Dissanayake, 
2009, p. 143) and «make the ordinary extraordinary» (Dissanayake, 2009, p. 143, see also Jakobson, 
1971; Mukagovsky, 1964; Shklovsky, 1917). In doing so, artists attract attention, maintain interest, 
and evoke emotions in their audience. 

To our knowledge, Joseph Campbell (1959) was the first to identify a relationship between art and 
the concept of supernormal stimulus, basing his assumptions on Carl Gustav Jung’s idea of “primary 
images”, namely archetypes (1921). According to Jung, there are two fundamentally different sys- 
tems of unconscious in the human being: the personal unconscious, on one hand, which is based on 
images derived from personal experiences, belonging to the unconscious memory; and the collec- 
tive unconscious, on the other hand, consisting of collective images, common to at least a group of 
people, which are found within us and which Jung calls “archetypes” or “primary images”. The 
concept of archetypes is akin to that of the “sign-stimuli” or “releasers” of instinctive behaviour 
identified by Lorenz (1950). The archetypes Jung refers to are a memory deposit, an engram, 
resulting from the synthesis of innumerable common experiences — a natural tendency with an 
anatomical and psychological basis. Speaking of art and supernormal stimuli, Campbell dwells on 
the environmental conditions in which the human brain has evolved and points out that our species 
for its first 600,000 years foraged for food and had to protect themselves and their families from 
turning into food for the very dangerous animal kingdom. Then, in the last 8,000 years, it has been 
serving comparatively safe farmers, merchants, and professors — figures characterised by societies 
capable of living off agriculture, trade, and culture. In Campbell’s view, there are structures in our 
nervous system that have remained unchanged for 600,000 years. The latter are indeed archetypical 
structures whose sign-stimuli or releasers are no longer present in our natural habitat but rather in 
certain artificial stimuli. According to Campbell, human culture has created an environment rich in 
artificial sign-stimuli capable of triggering automatic physiological responses, similar to those acti- 
vated in our distant ancestors. The author gives some examples of supernormal stimuli in human 
culture that would demonstrate their effectiveness in inducing concrete physical responses: repre- 
sentations of faces in prehistoric times, kingly robes, gladiatorial vestments, and cosmetics. These 
too, according to Campbell, could be explained by the presence of those previously mentioned 
“activators” that recall “inherited” images, i.e., archetypes, belonging to our past as humans in a 
natural habitat. It is worth mentioning that there are other hypotheses — which can complement that 
of Campbell - that have been provided to explain the forms and functions of the artefacts that 
Campbell reports as an example for illustrating his point. For instance, cultural influences and social 
needs play a major role today (De Block and Du Laing, 2010) -as they probably did in prehistoric 
times. Concepts such as that of ethnic markers — the signs by which ethnic boundaries are defined or 
maintained (Colman, 2015) — provide other perspectives contributing to the study of art, by focus- 
ing more on the cultural origin and societal necessities instead of the biological factors (Boyd and 
Richerson, 1987; Bell and Paegle, 2021). However, the role of culture in art is not in conflict with the 
idea of art as a supernormal stimulus, but rather it integrates it, highlighting, once again, the com- 
plexity of the debate around art. 

The concept of releasers is later taken up by Richard G. Coss (1968). He observes that, throughout 
the historical development of art, from prehistoric culture to the present day, similar decorative 
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elements (releasers) have been applied to human artefacts, which he identifies for instance in facial 
features, such as eyes, eyebrows, mouth, and teeth. The concept of the supernormal releaser (Coss, 
1968) could already be observed in the sensitivity to face patterns in objects made 75,000 years ago 
by Neanderthals, but also in the emphasised body forms of representations such as the Venus steato- 
pygia (Venus of Willendorf) of 30,000 years ago. 

Roy R. Behrens and Paul D. Whitson (1976), argued about art and the supernormal stimulus to 
explain the concept of art understood as mimesis, namely as imitation of reality. In their view, the art 
object, like the ethological supernormal stimulus, would be a more effective “representation” of 
reality than reality itself: the element defining the difference between the art object and the object in 
itself would be precisely the non-similarity, the difference from the starting model, which invites the 
spectators to awe and amplifies their reaction in front of the art object or the image. In The Power of 
Images. Studies in the History and Theory of Response (1989) David Freedberg tackles this topic by 
analysing the power that images have on humans in triggering reactions above the norm. Freedberg 
takes a new point of view in the analysis of visual culture, overturning the typical attitude with 
which images had been analysed up until then. Putting aside the traditional distinctions between 
artworks and “popular” images, between what is “beautiful” and what is “ugly”, Freedberg focuses on 
the study of the emotional reactions evoked by images (from sexual arousal to mystical contempla- 
tion) and the reasons underlying certain human behaviours towards images (the attribution of 
miracles, iconoclasm and the destruction of images). 

Aconnection between art and the concept of supernormal stimulus has also been drawn by Ferdinand 
Knobloch (1995/2000) for the specific field of music. According to Knobloch (1995/2000), music is 
able to elicit strong and deep passion in humans by stimulating the brain through the use of forms of 
supernormal stimuli that evoke evolutionary programmed releasers such as human male and female 
voices, laughs, cryings. In this view, musical instruments act by enhancing these types of biologically 
relevant stimuli (for alternative hypotheses on the evolutionary roots of music see also Fitch, 2016, 
and Haiduk and Fitch, 2022). 

At the end of the 20th century, the contribution of neuroscience to studies on art appraisal partly 
revolutionised the approach in this field, somewhat reaching — explicitly or otherwise — the idea that 
artistic stimuli actually act as supernormal stimuli. Neuroscientists such as Semir Zeki and Vilayanur 
Ramachandran have turned their attention to the neurobiological processes underlying art ap- 
praisal, through the contribution that neuroscience can make to the studies on psychic processes. For 
example, Zeki argues in Art and the Brain (1998) — a pivotal text for the definition of the new 
discipline of neuroaesthetics — that we should consider artists of the past as “natural scientists”. He 
suggests that artists were able to study the mechanisms of the brain and activate it in an aesthetically 
“valid” way, as they succeeded in abstracting the “essential characteristics” of an image, by eliminat- 
ing redundant information, and leveraging the neural mechanisms of perception. Further extending 
this perspective to the study of art fruition and aesthetic experience, Vilayanur Ramachandran and 
William Hirstein (1999) state that what the artist is trying to do — on a conscious or unconscious level 
— is to capture the essence of what they are representing and amplify it, in order to activate in an 
enhanced manner those neural mechanisms that would be activated by the object itself. Thereafter 
in the book The Emerging Mind (2003), Ramachandran identifies “ten laws of artistic experience”, 
eight of which were already mentioned in the first work with Hirstein (1999) — hyperbole, percep- 
tual grouping and binding, perceptual problem-solving, isolation, contrast, symmetry, abhorrence 
of coincidence/generic viewpoint, repetition, rhythm and orderliness, balance and metaphor — 
corresponding to certain perceptual mechanisms that use the amplification of perceptual character- 
istics of stimuli as a tool for art production. Authors posit that aesthetic appreciation rests on neuro- 
biological foundations akin to those of the peak shift effect — being, as above mentioned, a learning 
phenomenon that has the effect of shifting preference towards a specific object, which, in the case of 
art, would be, precisely, the artistic stimulus whose formal characteristics have been amplified by the 
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artist. The “caricatures” of archetype forms or images produced by artists act on neural circuits by 
activating them more effectively than natural stimuli, in fact acting as supernormal stimuli. Hence, 
the authors postulate the aphorism “all art is caricature” (not to be intended in a literal way) 
(Ramachandran and Hirstein, 1999; Ramachandran, 2003). 


Experimental Evidence Supporting the Vision of Art as a Supernormal Stimulus 


The field of neuropsychology, experimental aesthetics and more recently neuroaesthetics have 
offered different methodological, research and analysis tools for the study of aesthetic experience 
(Fechner, 1876; Freedberg, 1989; Ramachandran, 2003, 2004; Zeki, 2003). Since its inception, 
experimental aesthetics has sought to analyse the formal and phenomenological variables of art and, 
consequently, the cognitive and precognitive response of the public. The various schools of thought, 
from Fechner onwards, have played a decisive role in the debate on the phenomena of perception 
and their manifestation in the domain of art (see Jacobsen, 2010 for an overview). Aesthetics expe- 
rience and appreciation are influenced by several factors, such as evolutionary, anatomical or physi- 
ological constraints, but also culture, history, and individual differences (Jacobsen, 2010). In recent 
years, much experimental evidence has examined the properties of certain stimuli in the art field, 
validating their attributes as supernormal stimuli. These studies have mainly focused on the repre- 
sentation of the body in the artistic sphere, specifically on the level of attractiveness of the repre- 
sented forms of the human body and the viewer’s response to them (Thornhill and Grammer, 1999; 
Costa and Corazza, 2006; Doyle, 2009; Morris et al., 2013; Markovize, 2017; Pazhoohi et al., 2020a, 
2020b; Adam, 2021; Prokop, 2022). Some of these studies will be reported as a model of experimen- 
tal evidence supporting the idea that artistic stimuli act by enhancing the viewer’s reaction com- 
pared to the norm. It is important to make clear that for the purpose of this work we will almost 
exclusively focus on formal characteristics of artistic stimuli and the specific response of the beholder 
to them, to be used as an example of supernormal attributes. However, we are not neglecting the 
relevance of other factors, such as historical and socio-cultural ones, that are always intrinsic to 
artistic production and affect inevitably the aesthetic experience. 

Several studies in neuroaesthetics have highlighted how the representation of body parts and 
motor acts of human figures in certain works of art can activate our “mirror system” and induce 
cerebral activation to a greater extent than simple images (Ardizzi et al., 2020; Battaglia et al., 2011; 
Gallese and Freedberg, 2007; Gallese et al., 2021; Freedberg and Gallese, 2007). Neuroscience today 
refers to embodied cognition as the cognitive process that, applied to art, sheds light on how, when 
observing an artwork, not only vision is involved, but also the sensorimotor and somatosensory 
systems and the circuits that govern our ability to feel emotions (Gallese et al., 1996; Gallese and Di 
Dio, 2012). Quoting David Freedberg and Vittorio Gallese, the signs of which the artwork is 
composed would be «the visible traces of goal-directed movements; hence, they are capable of 
activating the relevant motor areas in the observer’s brain» (Freedberg and Gallese, 2007, p. 202). 
The authors show that the viewer when standing in front ofa painting reacts “as if” experiencing the 
action first-hand. This would be the case with figurative images, such as in Caravaggio’s Incredulity 
of Saint Thomas (1600-1601), where Thomas’ index finger “insistently” enters Jesus’ liver, but also in 
the case of abstract images, where the physical trace of the artist’s gesture is rather recognisable, for 
instance in Lucio Fontana cut canvases or in Jackson Pollock’s dripping (Gallese, 2009; Sbriscia- 
Fioretti et al., 2013; Umilta et al., 2012). In a transcranial magnetic stimulation (TMS) experiment, 
authors measured participants’ motor activity when confronted with Michelangelo’s Expulsion from 
Paradise (1510) and compared the observation of Michelangelo’s painting of Adam’s arm with a 
photograph of the same pose reproducing a human arm of the same size, and with the mental 
rehearsal of that painting (Battaglia et al., 2011). Results show that the level of corticomotor excit- 
ability recorded during observation of the photographic reproduction was significantly lower than 
both the observation of Michelangelo’s work and the corresponding mental rehearsal (Battaglia et 
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al., 2011). According to these data, the effect of observation of the artwork on the primary motor 
cortex is greater compared to that exerted by the photograph of a human arm (Battaglia et al., 2011), 
hence experimentally validating that art images can stimulate brain circuits in an enhanced manner 
as opposed to simple images, thus in line with the “supernormal” attribute. 

The latter study was chosen as an example to describe one of the possible approaches used in 
neuroaesthetics and, most importantly, it was selected precisely because it was one of the first, to our 
knowledge, to specifically compare the beholder’s response to a figurative scene depicted in an 
artwork by a worldwide recognised artist with a photographical control image depicting the same 
scene, which allowed us to make our point on the supernormal potential of artistic images. However, 
it is crucial to underline that there have been several other approaches — often less invasive than 
TMS, such as eye-movement analysis, electrodermal activity, heart rate variability, 
electroencephalography, event-related brain potentials (ERPs), magnetoencephalography, func- 
tional magnetic resonance imaging (fMRI) or positron emission tomography — that by analysing 
and correlating the beholder’s behavioural implicit and explicit responses have tried to define aes- 
thetic experience (for an overview see Jacobsen, 2010). 

The characteristics of Adam’s arm are in fact painted by Michelangelo in a way that they are 
amplified, exaggerated in some traits compared to the norm. As such, in keeping with the 
abovementioned study (Battaglia et al., 2011), they succeed in amplifying the neural response of the 
subject observing them. Many examples of this sort are witnessed throughout art history. A study by 
Costa and Corazza (2006) reported that portraits of faces share some characteristics across the 
different periods of art history: the roundness, width, and height of the eyes and the roundness and 
height of the lips are significantly more prominent in artistic portraits than in photographic ones; the 
width of the lips as well as lower-face roundness are, conversely, less prominent in artistic portraits as 
opposed to photographic portraits. Facial features including the shape and size of the eyes, lips, and 
faces are known to play a key role in the perception and processing of faces (see Haig, 1985; Bruce 
and Young, 1998) and so is their importance in artistic representations (see Koenig and Auge, 1975; 
Gombrich, 1982; Gregory et al., 1995). Feature saliency has been also attested by face scanning 
studies on eye movements (Gandelman, 1986), by studies that have drawn a connection between 
facial anthropometrics and the perception of beauty (McArthur and Apatow, 1984; Cunningham et 
al., 1990) and by studies that have shown that caricatured face images are more easily identifiable 
than images of original faces (Lee et al., 2000; Rhodes et al., 1987). Interestingly, at the neurological 
level, several experiments conducted with macaques have revealed an increased brain response in 
the areas responsible for face recognition when presented with exaggerated features (Freiwald et al., 
2009; Chang and Tsao, 2017; Kandel, 2012). Elements affecting dynamism, gait, and perception of 
movement, used as artistic devices, such as the contrapposto pose, have also been observed to have a 
more activating effect than the norm (Pazhoohi et al., 2020a). 

The studies reported here are by no means exhaustive of all the rich and diverse experimental 
evidence at disposal supporting the hypothesis that artistic stimuli are designed to be “superstimulating” 
to some extent. However, they do provide insights into the kind of approach neuroscience can share 
with research in the art field. 


Aesthetic Principles Outlining Forms of Supernormal Stimuli in 20th-Century Art Pro- 
duction 


Here we focus on the analysis of some of the aesthetic principles recently published by Numero 
Cromatico, an Italian collective of artists and researchers with a background in various disciplines — 
including neuroscience — which has been carrying out an interdisciplinary research for several years 
(Gagliardi, 2021). These principles are part ofa more detailed analysis previously published (Gagliardi, 
2021; Gagliardi et al., 2022). It is through such principles that we will be looking at the concept of 
supernormal stimulus in 20th-century art production, by seeking to stick to the two main levels that 
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we identified within the original ethological concept, the one of the amplified characteristics of the 
stimulus and that of the beholder’s reaction. 


Interaction 


When speaking of interaction, we refer to the relationship that the artist wants to establish with the 
viewer. In this sense, the artists design their works as real stimuli aimed at triggering a certain 
reaction in the public, which is as unprecedented, atypical and profound as possible. The principle of 
interaction, thus understood, has been incorporated in the art production of various artists and 
movements, especially since the advent of the historical avant-garde. Futurism, in fact, has overtly 
spoken of interaction (Marinetti, 1909, 1913), but after that, many have shown interest in the 
concept of interaction, by exploring it in different ways, from a purely sensory-body and a cogni- 
tive-conceptual perspective. Among these are the Russian Constructivists, Marcel Duchamp, Ex- 
perimental cinema, Arte Programmata, Kinetic art movements, those artists who delved into the 
radical zero-resetting of all expressive traits of painting in the late 1950s (including Piero Manzoni, 
Mimmo Rotella, Mario Schifano, Yves Klein, Sergio Lombardo), the artists of the famous event at 
Black Mountain College in 1952, Allan Kaprow and also Scuola di Piazza del Popolo, Pop Art, Visual 
and Concrete Poetry (Belloli, 1959), the Conceptual movement (Lewitt, 1967), in some cases also 
the Italian Arte Povera and certainly the Eventualist movement (Lombardo, 1987). 


Expressive Abstinence 


Recently formalised by Sergio Lombardo in his Eventualist theory (1987), the concept of expres- 
sive abstinence is part of a practice akin to various art movements, since the beginning of the 21st 
century. The aim of the artist embracing this aesthetic paradigm is to minimise to a great extent 
individual and subjective expression so as to favour the construction of aesthetic devices embedded 
with potentially destabilising, ambiguous, and polysemic characteristics, open to multiple interpre- 
tations, intended to stimulate a profound response from spectators. 


Minimalism 


Minimalism has been repeatedly taken up from different perspectives throughout the last century: 
as shrinking of the constituent elements of the artwork with Kazimir Malevich, who in 1915 painted 
a black square on a white background entitled Quadrangolo [Quadrangle], Spatialism (born in 1946), 
Robert Rauschenberg’s White Paintings dating back to 1951, Sergio Lombardo’s 1958 Monochromes, 
conceptual art from the second half of the 1960s up to Numero Cromatico. Minimalism means 
leaving no room for the superfluous, confusion, and arbitrariness in the process of designing an 
artwork (Gagliardi, 2021; Gagliardi et al., 2022). This approach to art research entails artworks that 
are enriched by formal components, materials, and techniques that fall within economic, ecological, 
and, above all, appropriate limits, specifically designed to “effectively” stimulate the public. 


Enriched Spaces 


Opting for enriched spaces is another trend that can be detected in the art production of many 
20th-century movements exploiting certain characteristics of the supernormal stimulus. It refers to 
the more general use of space, of the environment, as a form of multimodal aesthetic experience. 
Operating on the space of representation means using techniques, tools, and materials apt for trig- 
gering the public on both a bottom-up level (i-e., through the formal — namely material — character- 
istics of the artwork, which also implicitly activate the observer) and a top-down level (by stimulat- 
ing the observer’s reasoning, memories, and criticism) (Gagliardi, 2021; Gagliardi et al., 2022). The 
point is not staging something or artificially decorating the exhibition space to make it more attrac- 
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tive, but rather creating an installation that enriches the space so that an appropriate bond between 
the viewer and the artwork can be developed. The designated space hence results in a “scenario” 
[scenery], intended as a “constructed space” in which various elements (materials, surfaces, sounds, 
and smells) all contribute to establishing a profound and active relationship with the spectator. 


The Natural/ Artificial Relationship 


The use of natural and artificial elements (fabrics, pigments, elements from the natural world, 
technological devices, plastics, algorithms, artificial intelligence, texts, images, living beings, etc.) as 
supports, or devices, for artistic creation also fits within this framework. The challenge here is to 
strike a balance among all the formal elements while, simultaneously, achieving conceptual ambigu- 
ity in the artwork, one that stimulates the public to dwell on the inputs received in a way that is not 
artist-driven, but rather personal and open. 


Being Moved 


Lastly, due to its aesthetic, formal, and/or ritual characteristics, the artwork can elicit emotional 
states, which in itself is a form of amplified response of the individual. For several years neuroaesthetics 
has been investigating the so interesting concept of “being moved” (Kuehnast et al., 2014; 
Menninghaus et al., 2020; Wassiliwizky et al., 2015; Fiske et al., 2017; Zickfeld et al., 2019; Cullhed, 
2020; Mori and Iwanaga, 2021). The work of art can be understood as a device apt to “transport” the 
public aesthetically and emotionally into new, never-before-seen territories, challenging both per- 
sonal beliefs as well as those of entire cultures. Indeed, this principle is easily identifiable in art 
production throughout the history of art, not just that of the 20th century. 


Final Remarks 


The neuroscientific approach to the study of art has highlighted how our neural structures are 
activated to a greater extent by stimuli that present archetypical, enhanced, simplified, distorted, and 
symbolic forms. In this sense, the process of artistic creation would respond to needs peculiar to our 
neural systems. As pointed out by Zeki, owing to formal, perceptual, and conceptual knowledge, 
artists throughout history have succeeded — consciously or unconsciously — in condensing some of the 
mechanisms of perception in the creation of artefacts, environments, and experiences, often exagger- 
ated and unrealistic, yet capable of activating emotions and perceptual or cognitive phenomena in the 
audience that are more intense than reality itself. It follows that central themes of aesthetic experience 
such as beauty, emotions, perceptual ambiguity, the essential laws of nature, expression, and creativ- 
ity, are addressed by the artist by “superstimulating” the human brain. Far from covering all the 
available studies on the topic, this essay paves the way to further possible reflections on the view of art 
as a supernormal stimulus, so as to explain the neurobiological roots of the aesthetic experience, 
without undermining the relevance of socio-cultural and historical factors in these processes. In fact, 
the concept of supernormal stimulus is not only to be attributed to the formal characteristics of an 
artwork, but it could be also identified in a conceptual feature of it as well as in the general approach 
to it. We provided a few examples from the neuroscientific literature to better illustrate how the 
concept of supernormality can be found in formal aspects of artworks and to show how the beholder’s 
response can be measured in that specific case. Nevertheless, we also pointed out how the idea of 
supernormality is present in some aesthetic principles typical of 20th-century art production spe- 
cifically to highlight that this concept is generally embedded in the approach to artistic creation and 
it is inevitably linked to the contemporary context of the art of a specific epoch. 

We think that it is through such an approach that a bilateral exchange of knowledge and informa- 
tion between the artistic and scientific worlds is fostered; one that provides a mutual benefit to the 
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disciplines, thus contributing, on the one hand, to a better understanding of the mechanisms of the 
mind, and, on the other hand, to an understanding of art and the outlining ofits future horizons. 
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Aisthesis, Aesthetics and Cognition: Embodiments in 
Reading 


SARAH BRO TRASMUNDI & KARIN KUKKONEN# 


Abstract: The article argues for a renewed attention to “aisthesis” or the aesthetic dimension in sense 
perception in the study of reading, because cognition emerges from our embodied engagement 
with the world. As a case we focus on how three university students read Goethe’s Faust. We 
emphasise specifically how their readings differ significantly in relation to cognitive understanding 
and aesthetic experience. The embodied cognition framework has potential for changing the modus 
operandi in neuroscience, literary theory, and aesthetics by focusing on how understanding relates 
more broadly to aisthesis. We propose to investigate the link through readers’ aesthetic practices in 
relation to literary texts. 


Keywords: aisthesis, aesthetics, embodied reading, material engagement, Faust 


1. Aesthetics, Aisthesis, Embodiment 


Aw is generally understood as “a system of principles for the appreciation of the beautiful” 
(OED). Following the philosophical groundwork of Kant and Schelling, aesthetics outlines 
cultural practices that are related to largely exceptional experiences in music, art and literature. We 
listen to music and shudders run down our spine, we look at art and feel in touch with the endless, we 
read a literary text and get buoyed in streams of consciousness. Out of the three cultural practices 
described, listening to music, looking at art and reading a text, it is the last, reading, which does not 
belong to the aesthetic domain in a manner that is as obvious as in music and art appreciation. We 
read the newspaper, text messages, and sometimes literary texts. Reading is an everyday skill which 
can be put to the service of navigating the everyday world, as well as of appreciating the beautiful. 

If we approach the issue from this angle, suddenly also examples where other aesthetic experiences 
are embedded in the everyday come to mind: we listen to music on the metro, and we post beautiful 
images on our Instagram accounts. Luc Boltanski and Arnaud Esquerre observe an “enrichment” of 
our contemporary lifeworlds, where experience is sought out and charged with an aesthetic dimen- 
sion. The aftordances of digital social media, with wireless headphones and high-resolution cameras 
in our smartphones, arguably enhance this development. In reading literary texts, however, the 
boundaries between the everyday and the aesthetic are fluid and circulated to wide readerships 
already before the digital revolution, probably because letters on a page are more readily repro- 
duced en masse. Reading practices can seek information about the world or appreciation of the 
aesthetic, and they were under intense cultural discussion at least since the emergence of the (paper- 
based) public sphere and the cultural prominence of the novel in the eighteenth century (see Habermas; 
Littau). Since reading practices move back and forth between information and appreciation, we 
propose to deploy them as a case study for a yet understudied way in which aesthetics can take an 
important role in studying the human mind. 
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Here, the original meaning of “aesthetic”, namely, overall sensory perception or “aisthesis” (see 
Montani), and its later meaning in art appreciation comes into play. Kant’s period-making state- 
ments on the aesthetic in Kritik der Urteilskraft (1780) builds on earlier work by Alexander Baumgarten. 
Baumgarten writes Aesthetica in 1750, inventing the modern sense of the word “aesthetics” as taste 
judgement. He starts from the original meaning of the word “perception from one of the five senses” 
and extends it to refer to the ability to judge from sense perceptions, which we experience as pleasure 
or displeasure. Sense perceptions and the experience of pleasure and displeasure are understood as 
rooted in the body. Since Baumgarten, however, the embodied basis of aesthetics has moved into 
the background. “Taste” in aesthetics retains only a shallow metaphorical connexion with the origi- 
nal sense perception. Barbara Montero observes an overview article on “embodied aesthetics” that 
the “lower senses”, such as smell, taste and touch, are generally disregarded in modern aesthetics. In 
order to navigate the distance between subject and object, central to Kant’s aesthetics, contempla- 
tion at a distance is best achieved in the “higher senses” of vision and listening. 

In such a conception of aesthetics, reading is safely enshrined in visual perception. However, as 
recent approaches to reading processes and practices indicate, this conception misses a good part of 
how readers’ bodies are instrumental to accessing texts and to making sense of the written language 
and its narratives (Caracciolo and Kukkonen). Embodied narratology builds here on studies on how 
language and conceptual comprehension draws on embodied processes (see Gibbs; Barsalou; Zwaan 
for indicative references). When reading Goethe’s Faust, for example, multiple embodied dimen- 
sions are at play: motion verbs, indications of direction or metaphors evoke embodied resonances in 
readers and create thick situational models, especially when the language is formed for aesthetic 
appreciation. Mephistopheles introduces himself as “Ich bin der Geist, der stets verneint” (v. 1338; “I 
am the spirit of always saying no”, 46). His character as the negative of productive nature is specified 
by the light but repetitive iambus in the verse. (Read the German with the pattern da-dam-da-dam- 
da-dam-da-dam, and you get the effect). Also in prose, syntax and rhythm create similar effects of 
embodied pacing (see Kreiner and Eviatar). Mere visual perception does not account for embodied 
pacing. Indeed, it is the embodied cognition (usually placed lower on the aesthetic hierarchy) that 
stands behind the aesthetic appreciation here. 

Aisthesis (whole-bodied sense-perception), aesthetics (experiential appreciation) and higher cog- 
nitive processes (understanding) have historically been studied in isolation, but the framework un- 
derlying distributed and embodied cognition (Hutchins) allows for integrating these strands. Fur- 
ther, Noé (xi) suggests that life and art are entangled, preconditions for one another, that is: “We are 
aesthetic phenomena. To understand and know ourselves, we need to undertake an aesthetic inves- 
tigation of that work-in-progress that is the self we are.” Being an aesthetic human means that one’s 
embodied engagement with the world is always constrained by habits, values, materiality, technol- 
ogy, and biology. For instance, of all the possible ways an object, ora situation, could be attended to, 
the actual perception is organised and structured by a system much larger than that of the situated 
individual (see Goodwin, “Professional Vision”). Appreciation of an object or a situation is there- 
fore, a distributed cognitive-perceptual task shaped by habitual forms of attending with actual 
physical properties of the situated (Hutchins; Noé). This idea explains why persons who have learned 
to read automatically perceive symbols or words, when written marks are combined in certain ways 
and contexts. Together, all these bio-social mechanisms of perception shape what we attend to, and 
thus in some sense mould appreciation (Gibson). This distributed mechanism also impacts how 
reading is organised, performed, and appreciated; an issue that is especially salient in today’s digitised 
reading environments (see Mangen, Olivier and Velay). AsAnezka Kuzmiéova puts it in the title to 
an article, it can “make a difference where you read” and the reading environment can be much 
more than a simple distraction from the text at hand. 

While the social values related to appreciation might be constrained by human biological archi- 
tecture (e.g. the universal preference for symmetry), a biological or evolutionary account of appre- 
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ciation in isolation will not suffice. Aesthetic experiences are thus not only a result of direct sensory 
inputs by the nervous systems, they are also enshrined in cultural cognition and reading technologies 
that come with their own cultural evaluations. In other words, appreciation is dependent on social 
judgment and must therefore be based in ontogenetic development. Its cultural roots enable hetero- 
geneity, aesthetic variations across cultures, that are passed on by ancestors and learned through 
moral, educational, and linguistic highlighting (Goodwin “Professional Vision” and “Co-Opera- 
tive”). The ontogenetic development has ties to broad cultural practices but also individual life 
processes that is observed as a bodily, subjective style. In this article, we are thus interested in how the 
cross-section between social cultural values, individuation, and material engagement with literature 
enables cognitive-aesthetic experiences. 

Human aisthesis involves not just local perception of what is there, Trasmundi and Cowley argue, 


Specifically, aisthesis arises as one gains experience of attending to one’s engaging with material 
properties of the world. Asa result, people develop expertise of sensibility. In the case of reading [...] it 
depends on timing how we saccade (and move) while drawing on expectations and feelings. In that it 
is prereflective, one cannot set out to explain why it happens or what it means; one can only track 
evidence for its occurrence. It plainly includes echoes of previous seeing, hearing, smelling, tasting, and 
touching. Aisthesis thus draws on continuous prereflective judgments that arise in local engagement 
with visible patterns on a page. The resulting ways of looking feeling and, in some cases, vocalizing are 
constrained by how one draws on expectations, emergent properties of the situation, the tools with 
which one is engaged and the historicity of the engaging body, in this case as a reader. (2) 


In contrast to non-primates, humans preserve texts for aesthetic purposes and they engage with them 
in ways that change their perspectives on themselves. Written materials allows people not only to 
transfer information across time, but also to explore how learned ways of attending affect judge- 
ments and experiences. 

The tight relationship between embodied cognition and art appreciation has not gone unnoticed 
in the study of music and the visual arts (see Hoffding, Sanchez and Roald). However, it may be the 
study of literature where it can be investigated with particular precision, because reading unfolds in 
contexts that are both information-driven and appreciation-driven. In this article, we propose to 
outline how embodied, aesthetic reading practices could be investigated and analysed empirically, 
based on a small “proof of concept” study. We show how readings of Goethe’s Faust reveal cultural 
techniques of material engagement in ways that impact aesthetic reflection and understanding. We 
argue that this engagement is characterised by an oscillation between continuous aesthetic and 
information-based judgments and does not necessarily depend on the primacy of the one over the 
other. By that we mean that, usually, reading and education research prioritises an information- 
based approach to reading, which emphasises word- and sentence-level comprehension and other 
cognitive dimensions, such as memory and analytical interpretation (see for example the handbook 
The Science of Reading; Hulme, Snowling and Nation). The question arises what role appreciation- 
based reading is assigned in reading and education research, ifinformation-based reading is consid- 
ered primary. The aesthetic dimension in reading is often tied to the analysis of literary texts and 
their aesthetic or “poetic function” (see Jakobson), even in the neurosciences (see Jacobs), or to 
investigating how readers conceive of the aesthetic through surveying the verbal concepts they use 
(Knoop et al.). We question whether the information-based approach has functional supremacy 
over the appreciation-based approach and propose to investigate the matter primarily through 
readers’ aesthetic practices and how they relate to the aesthetic qualities of literary texts or readers’ 
aesthetic judgements. 

The empirical study that serves as the basis for this article focuses on a small number of readers in 
order to provide a multi-faceted analysis of information- and appreciation-related aspects in read- 
ing, as well as their highly personalised nature. We aim for unique insights over broad empirical 
generalisability, proposing our analysis as “proof of concept” for future, more systematic investiga- 
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tions across a range of texts. The study is designed on the basis of ethnographic work to illustrate how 
cognitive-aesthetic judgements in reading can develop, but it also draws on methods from reading 
research and cognitive literary studies. The aim is thus to show how, in different contexts, readers 
engage with texts and use embodied strategies to control their cognitive-aesthetic processes during 
reading. 


2. A Cognitive Video-Ethnography of Students’ Reading of Faust 


The aesthetic and cognitive dimensions in embodied material engagement, such as reading, are 
often temporally intertwined. We exemplify this point with examples from a cognitive video- 
ethnography study of three university students’ readings of Goethe’s Faust. The cognitive ethnog- 
raphy framework (Hutchins; Trasmundi) invites multi-method investigations of their actual read- 
ing practices, and this method can be used to observe, engage and converse with the readers, but also 
as a means to explore their immediate observable judgements and their analytical reflection of how 
they experience the text during and after reading. Hutchins coined the term ‘cognitive ethnography’ 
and argued for the need to study human cognitive processes outside the laboratory. He defines the 
method and its significance in his seminal work, Cognition in the Wild in these words: 


Many of the foundational problems of cognitive science are consequences of our ignorance of the 
nature of cognition in the wild [...] The first part of the job is, therefore, a descriptive enterprise. I call 
this description of the cognitive task world a “cognitive ethnography.” One might have assumed that 
cognitive anthropology would have made this sort of work its centerpiece. It has not. Studying 
cognition in the wild is difficult, and the outcomes are uncertain. (370-371) 


In the cases we present below, we seek to bridge the cognitive divide between information and 
appreciation-based approaches as opposite and layered phenomena by applying cognitive ethnog- 
raphy in innovative ways within this domain. Therefore, we scrutinise how the processes can over- 
lap and emerge as non-linear forms of reading. As we shall see, understanding has an aesthetic basis, 
and the appreciation-based approach can also bring forth functional outcomes and enhance under- 
standing of the information presented in the text. 

Three qualitative case examples from the study are presented below. The defined task and the 
socio-cultural frame are the same for all three students: they all read the text as a preparation for the 
same class, and they all articulate the purpose of their reading as being able to understand, remember 
and reflect on the text using analytical tools from the curriculum. Yet, they all read in very different 
ways that lead to significantly different experiences and understandings. As we will show, the stu- 
dents enact very different cognitive-aesthetic strategies. Some of the strategies were more func- 
tional than others. Below we describe each reader-setting before we analyse the characterises of 
different (dys)functional cognitive-aesthetic strategies (see section 3). 


2.1. Setting the Scene: Readers and their Preparatory Reading 


According to the state of the art in cognitive science, cognition is — as mentioned above — distrib- 
uted, observable and grounded in natural actions of behavioural coordination (Hutchins). Follow- 
ing Hutchins’ suggestion, grounding cognitive research in the domain of natural, real-life cognitive 
events, makes one immediately aware of the methodological challenges that follow such a decision. 
To exchange the laboratory setting for a natural habitat of a sociocultural practice entails consider- 
able repercussions for the way the research design should be framed (Hutchins). Cognitive ethno- 
graphic fieldwork is thus suitable for generating thick descriptions (Geertz) as part of the investigation 
of what happens in a socio-cultural context. We thus prioritised a richer level of details and such 
thick descriptions to get beyond general, broad conceptions of reading. Particularly, three students 
were recruited for this study'. One male and two female students in their early 20s. The participants 
voluntarily signed up for the study. They were informed that the researchers had a broad interest in 
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getting a better grip on the bodily processes in reading and how their body and reading was expe- 
rienced by the readers themselves. The three university students were all doing their MA in German, 
and they were all part of the same study program. The reading cases we present here comes from 
their preparatory reading in relation to a course in modern, German literature. We will refer to 
them as reader 1, 2 and 3 respectively (see Figure 1 below). Reader 1 is a native German speaker, 
whereas reader 2 and 3 are native Danish speakers, but enrolled in a master programme in German. 
All students read in their natural, self-preferred settings. The medium on which they read, is also 
self-chosen, and depended on which version they had bought for the particular course. Two students 
preferred the paper book, whereas the third downloaded the text on his computer and read it on the 
screen. They read different editions. They all notified the observer about when and where they 
would read, so that the observer - prior to the reading — could set up cameras. The cognitive 
ethnographic fieldwork underlying this study, involves a combination of methods ranging from 
observational research such as video-observation and participant observation as well as observing 
material culture over time and collecting qualitative interviews. The design allows for a dual focus 
on both the micro and macro scale of reading, as well as a registration of an experienced and 
observed level of behaviour and sense-making. Specifically, the observer was present and took 
fieldnotes during their actual readings in those locations”. Afterwards the observer conducted semi- 
structured interviews with the readers, and asked them about their reading experiences — both their 
concrete and general experiences — to learn about their reading habits, strategies, and personalities. 
In total, the dataset used for this analysis, counts approximately three hours video-observation of 
their reading and also three hours interviews. The data were coded by use of the method of Cognitive 
Event Analysis (see Steftensen) that is developed to study how humans adapt to and modify their 
environment in order to achieve a task. Space limitations prohibit a thorough unfolding of the 
analytical method (for a detailed presentation, see Trasmundi, Steffensen). In this article we highlight 
those moments that has significance for the aesthetic and cognitive-affective reading experiences. 
Below is an overview of the three observed reading settings, as well as the materials they had at hand. 


Figure 1: Overview of three readers and their preferred reading settings 
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Below follows an elaborate description of each reader’s reading setting. The description is crucial 
for how the aesthetic-cognitive processes emerge as we emphasise in the analysis. 


2.1.1. Reader 1: Reading at Home at the Desk 


Figure 2: Reader 1 


Reader 1 reads at home at her study table, which is her general, preferred reading location. Prior 
to the reading, she makes the setting comfortable, and she gets blankets, lights a candle, gets her 
teapot and she adjusts the lightning in the room. She puts herself in a casual position on the chair, as 
visualised in the picture above. She reads a printed book version of Faust. It is her own book, which 
she has read before in high school. Her reading setting is designed so it affords a comfortable and 
flexible, multimodal reading mode that prompts a soothing emotional threshold. At the same time, 
large part of her reading is centred around seeking information. She has secondary texts lined up, just 
as she has organised pens and rulers in front of her at the table. She further explains that the aim is to 
read, understand and be able to discuss the intentions, interactions and functions of the characters in 
the story. For instance, she must be able to recall and explain what happens in the story world when 
Mephistopheles appears etc., and for that reason, she has organised the setting in a way that scaffolds 
the realisation of this task in efficient and predictable ways. Overall, her reading setting reveals her 
role as a student preparing for an information-based reading task. 


2.1.2. Reader 2: Reading at the University in a Lecture Room 


Figure 3: Reader 2 
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Reader 2 reads at the university, in a lecture room and on a computer screen, even though he has the 
printed book version on the table too. Like reader 1, he has prepared the reading setting, and has 
brought pastries, a soft drink, coffee, and he has direct access to both pen and paper. In contrast to reader 
1, reader 2 wears headphones throughout the entire reading as he listens to music on his cell phone. 

His understanding of comfortable and effective reading is different from reader 1. He places 
himself in a public space which invites a different form of reading attitude compared to a homely 
setting. The different locations might reflect their different personality traits as reader 2 explains, 
that he needs to place himself in an environment that constantly reminds him of the task and de- 
mands during reading. Further, he reduces auditory couplings to his environment through his use of 
headphones, whereas reader 1 prefers complete silence. 


2.1.3. Reader 3: Reading at Home on the Floor 


Figure 4: Reader 3 


The second female reader, reader 3, reads at home. She lies on the floor on a yoga mat in her own 
apartment and reads in a print book. She has her computer next to her in case she needs to look up 
and explore a concept in the text further, she explains. Her reading setting is characterised as very 
personal and intimate and in ways that enable the body to be more relaxed compared to table 
reading. Comfort is much related to the bodily flexibility, whereas readers 1 and 2 were more 
concerned with generating comfort through other factors, such as food, soft drinks/tea and candles. 
Reader 3 only has a bottle of water next to her and a little reading lamp that provides her with 
optimal visual conditions for reading. 


3. Analysing Cognitive-Aesthetic Processes in Student’s Reading of Faust 


As mentioned above, the students read Johann Wolfgang von Goethe’s Faust. The text is a touch- 
stone of modern literature. The protagonist seeks knowledge and the key to lead a better life, so 
much that he is ready to strike a deal with the devil for these ends, but he fails to understand the 
human needs of the woman he loves. Goethe entwines metaphysical and psychological matters in 
this drama, and he brings key controversies of the modern world into dramatically compressed 
situations and highly recognisable lines. While Faust is a drama, and the verse lines are allocated to 
individual speakers who are then realised by actors on the stage, such as Mephistopheles famously by 
the actor Gustaf Griindgens, it is -today- mostly read by pupils and students in German literature. 

Weare interested in studying how the three readers enact different cognitive-aesthetic reading 
strategies when they engage with this text and genre. Further, we want to explore how those strate- 
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gies are bodily enacted, experienced and how they function in relation to the task. Based on obser- 
vation of their preparatory reading (which in this case lasted approximately one hour), we present 
four overall embodied reading strategies identified in the dataset across the three readers. The 
strategies amount to: (i) drawing in reading, (ii) writing in reading, (iii) bodily coordination, and 
(iv) rendering out loud. The strategies are analysed in relation to their aesthetic-cognitive function 
and experiential value. 

We underline that the aim of the analysis is not to suggest general or universal functions or to claim 
an exhaustive list of embodied strategies, rather, we point to actual particular cases that might lead to 
more systematic investigations of how embodied strategies function for aesthetic-cognitive reading 
more generally. 


3.1. Drawing and Writing as Part of Reading: Aesthetic Marking as Ordering Information 


The first embodied strategy, we explore, is that of drawing while reading. The phenomenon of 
drawing was a rare exception that only happened two times in this preparatory reading study. While 
the function of drawing proves to be an aesthetically and cognitively significant activity for under- 
standing, writing is a much more profound means for ordering information. Regardless of the kind 
of trace-making (drawing or writing), both types of imprints are judged aesthetically, affectively 
and cognitively. For instance, handwriting is unique and personal. Handwritten concepts (words 
and sentences) thus carry a unique bodily trait, because every handwriting is different and reflects a 
person’s control of the micro movements and coordination (Chemero). Those traits are judged 
more or less beautiful according to both cultural and personal style (Ingold). Likewise, the visual 
features enabled by design and use of pencils and ball pens leaves very different imprints even though 
they may signify the same linguistic meaning. The fragility, manifestation, colour, and other mate- 
rial affordances of different markers impact the aesthetic perception, and even memory (cf. 
Malafouris). 

Interestingly, while both reader 1 and 3 make multiple handwritten marks during their reading 
(highlighting, underlining, note-taking, drawing), reader 2 does not take any notes at all. Recent 
results from reading research on text materiality suggest that screen reading affords less manual 
interaction with the text (Mangen; Mangen and Van der Weel). Further, handwriting is a non- 
uniform aesthetic trace, whereas computer generated notes have a universal, mechanical and pre- 
dictable expression. The disembodied and predictable expression affects appreciation. We thus 
explore these effects by asking what the aesthetic-cognitive value of drawing and handwritten notes 
is in the cases observed in reader 1 and 3. 


3.1.1. The Aesthetics of Drawing Notes: Visualising and Anchoring Information 


In most cases, reader 1 and 3’s handwritten note-taking amounts to alphabetic annotations in the 
margins. However, in one instance both readers make use of drawing. The reason might be that, 
generally, drawing is a cognitively expensive, time-consuming, and skill-demanding activity. Writing 
is simply easier to manage. In informational terms, a translation from one modality to another (from 
verbal to drawing) do not add to the meaning of a symbol. It adds however an aesthetic dimension 
to the narrative and makes the reading experience richer and more tangible for the reader. It is a 
striking fact that the only drawing that reader 1 and 3 make during the 1-hour reading happens to 
be the same, namely ‘a flame’. This circumstance indicates a deeper cognitive challenge related to 
the flame-contexts. When reader 3 was asked why she only once made use of another marking 
technique, she hesitated. In the attempt to reconstruct the reason for the drawing, she mentioned that 
she “just felt some sort of importance of the word ‘flame”. The pictorial manifestation of the flame 
gave it a certain hierarchical priority compared to the multiple alphabetic notes. As such she re- 
sponded to the feeling by giving it a unique aesthetic-cognitive trace in the text. 
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Reader 1 mentioned that ‘the flame’ in its context “was an interesting image,” and that the drawing 
“just happened”. While both readers respond in similar ways (by drawing) to the intriguing signifi- 
cance of ‘the flame’, the contexts, in which the flames appear, are not the same. Likewise, their 
drawings also differ slightly, but might serve similar aesthetic-cognitive functions at a basic level. 
Reader 3 highlights the stage direction: DER GEIST erscheint in der Flamme (16; English trans- 
lation): “The SPIRIT appears in the flame” (20). See drawing in the picture below. 


Figure 5: The flame drawing in the left margin, made by reader 3 


In the left margin, reader 3 has drawn the flame and an abstract contour of the SPIRIT rising from 
the flame, almost as if following the stage directions herself. Reader 1, on the other hand, draws ‘a 
strainer and a flame’ in continuation of the stage direction: Sie fahrt mit dem Schaumléffel in den 
Kessel und spritzt Flammen nach Faust, Mephistopheles und den Tieren (71; English transla- 
tion): “She thrusts the stirring spoon into the cauldron and splashes flames at Faust, 
Mephistopheles and the animals” (86). Reader 1 draws the following picture. 


Figure 6: The flame drawing by reader 1 


The flames belong to the alchemical activities of Faust (in the first instance) and to the witches’ magic 
(in the second instance). They relate the supernatural in the play to something that is visually percep- 
tible. Readers 1 and 3 make for themselves visually perceptible what remains printed words in the 
stage directions. Their drawings become aesthetic markers that organise understanding around the 
theme of the supernatural. Adding a pictorial dimension to the description can help highlighting the 
information as a richer, more embodied perception, and the “flame” is an emotionally salient, atten- 
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tion-grabbing image (see Forceville). The drawing thus becomes the imagery correlate to the 
supernatural textual description which can enhance memory, and make the information more tan- 
gible and concrete, as the empirical correlate to flame-throwing and spirit-rising are absent. 

It seems that drawing can give the fantastic or empirically unreal a shape of reality — an appear- 
ance. The drawing might serve as a functional aesthetic-cognitive strategy that enhance memory, 
emotional engagement and even understanding as one is forced to think the scenario through in the 
concrete. However, while drawing is one kind of (rare) embodied trace-making during reading, 
handwriting is much more profound. 


3.1.2 .The Aesthetics of Handwritten Notes: Re-Arranging Information 


Both readers 1 and 3 make multiple handwritten notes in the margin. Their annotations often are 
copies of sentences in the text. Other times they take shape in forms of questions or interpretations. 
Those cognitive, repetitive aspects of their writing help sediment their memory, or they allow them 
to extend their thinking and elaborate on the information in text (adding their own or other per- 
spectives to what is there). The notetaking, in itself, reveals an information-based approach, but 
especially reader 3, is also concerned about the aesthetic traces her writing plants. 

The very mark-making and the tools reader 3 uses to take notes are important choices that impact 
her reading experience. We observed how the book she reads already had multiple notations made 
by a ballpoint, and when we talked to her about it, she felt almost ashamed and highly distanced from 
this kind of mark-making. She reveals: “I used this book in high school too, and I just realise I wrote 
with a ballpoint. | would never do that anymore. It looks really terrible — to see the marks in the book, 
but I was probably not thinking about that back then.” 


Figure 7: Annotations by reader 1 


As she utters ‘never’ she makes a significant gesture with both hands (see the picture above) and 
laughs. Her laugh, gesture and utterance of ‘never’ reveals the emotional dissonance between her 
former and current readerly self: The symbolisations that constitute the narrative in the text are one 
set of experiential anchors, whereas self-produced marks over time (then and now) give insight into 
skilled reading, i.e., the kind of link between individual experience and conventionalised practice of 
judging the aestheticisation of marks. 

Reader 3 has decorated her workspace with plants and colourful garlands, giving it an aesthetic 
dimension. Arguably, she chooses her writing implements with similar considerations. Indeed, de- 
spite the ongoing digitisation of learning environments, craft pens and biros, as well as high-quality 
notebooks appear still to be in high demand. Arguably, this links to the overall phenomenon of the 
“aesthetisation of the world” (Lipovetsky and Serroy) or to “enrichment” (Boltanski and Esquerre), 
but, on the level of reading practices, these aesthetic implements also support the individuation of the 
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reader as well as the refinement of their aesthetic judgements. Reader identity becomes traceable not 
only in a single act of reading, as readers follow the written record, but also across the aesthetic 
aspects of the note-taking at different points in a reader's biography. 


3.2. Bodily Rhythm and Text Flow: Cognitive-Aesthetic Resonance and Dissonance 


The recently developed scale for reading flow RFSS (Thissen, Menninghaus and Schlotz) identi- 
fies the dimensions of presence (that is, the feeling of being there in fictional worlds), identification, 
suspense and cognitive mastery. It stays exclusively with what is written in the text, but not how that 
text is read. Flow, in reading, is enabled by a strong coupling between reader and text and analysed 
as the reader’s ability to engage with a text’s stylistic features and structure (see also Kukkonen). This 
coupling can be enhanced and nurtured by creating the best conditions for such engagement, just as 
it can be inhibited by external disturbances that break the engagement and inhibit focus, concentra- 
tion and even rhythmic sensitivity. We underline, that depending on the text and purpose, breaking 
the flow — or decoupling - can be crucial for learning, critical thinking and imagining (cf. Trasmundi 
and Toro; Trasmundi et al). Yet, the crucial question is whether the (break of) flow is elicited by the 
reader herself, or whether it is caused by inability to concentrate focus and engage due to external 
stress imposed on the reader-text system. Below we describe how reader 2’s reading setting forces 
him into fixation loops that might be detrimental for his reading experience and sense of being ina 
good reading flow, exactly because he is caught between being coupled with the rhythm of the text, 
and the rhythm of the music he listens to (see also Figure 8 below). 


Figure 8: Reader 2 listening to music 


Reader 2 reveals many embodied signs of frustrations throughout his 1-hour reading. From the 
videos, he systematically sighs, he shakes his head in resignation, and he moves uncomfortable on his 
chair. Further, he raises his eyebrows and then seeks for something in one of the books in front of 
him, and resumes reading without being able to reduce the emotional tension that his body reveals. 
In between these embodied shifts, he gets into a musical flow and uses his hands and upper body to 
enact a beat rhythm. He appears to be more strongly coupled with the music than with the narrative. 
Indeed, his reading flow is constrained by the musical rhythmicality more than of the text rhythm 
constructed by stylistic choices (such as prosodic features, syntax and word/sentence length). 
Reader 2’s frustration was also addressed in the interview after the reading. He uttered that he 
found the text difficult and that there were several words that he struggled with. While we cannot say 
directly how his musical absorption relates to his struggling, it is plausible to assume that his reading 
environment inhibits flexible embodiment and thereby turns into an obstacle for text-flow. He 
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breaks the flow and repeats the reading, and thereby experiences a struggle. There is no way to 
predict whether another strategy will remedy the struggling he experiences, yet his options are kept 
at a minimum, and his embodiments are constrained. In contrast, we see, below, a much more 
flexible, adaptive reading trajectory enacted by reader 3, who runs the reading through her body in 
multiple ways that seem to be conducive for her reading flow and experience. 


3.3, Rendering out Loud: Running it through the Body 


Reader 3 also explains how she struggles with understanding metaphorical expressions and also 
translation issues during her reading. However, observations of her engagement with the text re- 
vealed several flexible strategies. First, she lies on the floor, and eventually she enacts a particular 
thythm by moving her shins back and forth. Compared to reader 2, reader 3 does not listen to music. 
From the interviews, she emphasises that she likes to lie on the floor, because her body does not 
become exhausted and tired from sitting up straight, and it is possible for her to move more easily 
when she seeks to sustain attention, for instance. Sometimes her movement mirrors a tempo or 
reading rhythm, for instance when she pronounces a difficult word, her legs movements will be 
aligned with her vocal articulations of syllables. Yet, her rhythmic movements are not always con- 
ditioned by the stylistic features in the text, rather, she explains, she can also impose and invoke 
thythm through bodily coordination so she will maintain focus and force a flow onto the text, which 
enables her to maintain a constant pace. Reading is running through her body and involves both 
brain, eyes, hands, torso, and legs. 


‘i 


} 


However, we also observed another bodily strategy that she used, when she experienced both de- 
cline in attention span, and when she felt unfamiliar with peculiar words. One way to get familiar 
with expressions or words is by rendering out loud to simply taste them in a rich multimodal sense. 
Particularly, we observed how she eventually, would switch between ‘silent’ reading, to silent read- 
ing where she moves her lips, but without producing any sound, to a soft whispery reading and 
finally to reading out loud. When asked why she did that, she gave reference to both aesthetic and 
cognitive explanations. Starting first with the functional explanation that matched the emotional 
frustration that relates to attention span, she uttered: 
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“I do it because it is another language. [...] the fact that I hear it — the case that it is not just a voice 
inside my head, but | actually also hear it, allows me to focus my attention; that my thoughts so to 
speak are moved in the background [...] but I could also do it in Danish, because I like to use multiple 
senses in some way to.., well then that is the only thing | need to focus on, and I am not going to 
remember what I will be doing tomorrow [...] which I often do when I read inside my head” 


Figure 10: Reader 3 as she utters “moved in the background” 


As she explains how she seeks to “move the thoughts in the background” she gestures (see Figure 10) 
to underline how her strategy allows her reading to be foregrounded and disturbances are left 
behind. Her voice materialises and speaks to multiple senses at the same time, and it constrains 
attention very efficiently. However, she also mentions that at other times, she simply prioritises the 
aesthetic dimension of reading over information. Or in other words, she shows how the aesthetic is 
intertwined with understanding. She elaborates: 


“I think German is awesome, | enjoy speaking it. So, therefore it is also. .. it is just cool I think. To read 
it out loud and hear it, and at the same time feel it in the mouth, how the words kind of feel.” 


Figure 11: Reader 3 as she utters “feel it in the mouth” 
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Producing music through reading aloud is not just ‘auditory cheesecake’ (Pinker). Rather, the aes- 
thetic and functional are inseparable aspects of reading experience. And to “feel it in the mouth” is 
another way of running the reading through the body. Reading is a highly embodied activity that 
meshes aesthetic sensations with understanding. 

The examples in the analysis above serve to guide reading and literary research in more embodied 
and aesthetic-based directions. We suggest that readers exploit multiple embodied strategies when 
they read, because exploiting the body’s multimodal potential, can be a rewarding strategy in terms 
of aesthetically-cognitive outcome and experience. 


4. Limitations 


Our study highlights tendencies in readers and identifies areas warranting further investigation, 
along with proposals for theoretical backdrops to inform the analysis. We fully acknowledge that 
there are inherent limitations in the kinds of generalisations we derive from a relatively limited 
number of subjects and recordings. In our research, we undertake a qualitative investigation of the 
ecological “wild”, where “studying cognition [...] is difficult, and the outcomes are uncertain” 
(Hutchins, 371). The limitations are clear as our study does not yield the “pure” and generalisable 
results achieved from quantitative research (see Dreier). Quantitative analyses may elucidate gen- 
eral tendencies, but are detached from social practices, and therefore lack the explanatory power 
needed to understand the unique coordination intrinsic in ecosystems of cultural practices. 

In foregrounding the interaction between aesthetic and cognitive dimensions of reading, we 
demonstrate how these events are managed in situ. As an initial step towards highlighting this 
interaction, our study is limited, but it can link to larger quantitative studies, investigating a single 
aspect on the one hand, and future inquiries into how cultural-material constraints—such as media, 
genre, norms, texture, etc.—affect these cognitive processes more broadly, on the other hand. 


5. Conclusion 


Why read Goethe’s Faust in German classes? A traditional argument would perhaps run along the 
lines of mastering the German language in its most complex realisation or acquiring fundamentals of 
cultural capital in German literature. Our case study suggests another set of reasons building on an 
embodied understanding of reading — such texts inspire students to explore the aesthetic dimension 
of reading in a foreign language and thereby have the potential to bypass hurdles in reading com- 
prehension. Reader 3, for example, experiences German as a cool and enjoyable language in a strongly 
embodied mode, which arguably motivates and keeps sustainable her engagement with this difficult 
text. We have seen similar effects in the other readers’ aesthetic practices, as well as difficulties if the 
aesthetic dimension is shut out. Our study involved only a small number of participants, but we hope 
that it can serve as a “proof of concept” for how appreciation- and information-driven reading 
processes can feed off one another. We have shown how readers use the aesthetic — appreciation- 
based approach to get into the flow and engage with the information in efficient ways. Sometimes, 
the approach can be very specific: one searches for meaning, or one enjoys a sound, but at other 
times the cognitive and aesthetic intermesh more globally in the reading setting, involving pen and 
paper or alternative aesthetic inputs, such as music. 

The small-scale explorative study we presented here indicates numerous lines for the empirical 
analysis of aisthesis in reading could proceed: 


(1) Reading aloud vs. reading silently. Reading a literary text, written in verse, can contribute 
to students’ understanding of a text and, moreover, to inhabiting a foreign language both 
linguistically and emotionally. Reading research has foregrounded the importance of the 
“phonological route” for sounding out words in beginning readers (Dehaene), but clearly the 
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aesthetic dimension of reading aloud has much untapped potential for reading experiences 
and it could profit from insights on the effects of prosody, verse and metre in literary studies. 


(2) Flow reading. The study of reading flow foregrounds textual elements that improve or 
impede “flow” (see Thissen, Menninghaus and Schlotz). Flow, however, has a strongly em- 
bodied dimension (see Pianzola; Kukkonen) that arguably is also affected by the readers’ 
physical position and by the configuration of the environment they find themselves in (see 
Kuzmiéova ). New directions in “ecologically valid” reading studies could investigate the 
interrelation between the relative importance of flow in embodied language and flow in 
readers’ bodies. 


(3) Interactions between multiple modes of aisthesis. Readers in this study chose to write on 
their texts and draw images, they listened to music and decorated their reading environ- 
ments. Predilections in verbal, visual, and musical mental imagery differs between individu- 
als with different “cognitive styles” (Kozhevnikov). How do readers enrich the written text to 
suit their own predilections, as a kind of self-designed “environmental propping” (Kuzmiéova)? 
And could this enhance the study of different modes of mental imagery involved in reading, 
which so far has privileged visual mental imagery? 


Lipovetsky and Serroy are concerned about new antinomies between an aesthetisation of the world 
on the one hand and new forms of “losing ourselves” (“Nouvelles forms de dépossession subjective” 
466) in measuring our body stats, medicalising what we eat and what we do, prizing efficiency 
above all, overemphasising the virtual, and engaging in hyperconsumption. At the end of their 
book, they express hope that society can move past these antinomies (482). The readers in our 
exploratory study designed practices between information and appreciation through an engage- 
ment with the aesthetic dimension of the reading process, arguably not “losing themselves” but 
finding reading strategies that suited and satisfied them aesthetically. Our embodied starting point 
encourages work across disciplinary boundaries and dialogue with neuroscience, phenomenology, 
cognitive science, linguistics, anthropology and literary studies. Indeed, literary studies may take the 
lead in such an endeavour because of its long-standing attention to appreciation- over information- 
driven modes of reading. From such an interdisciplinary investigation may come not only a better 
understanding of reading as an information- and appreciation-driven process but also possible 
teaching practices to encourage young readers to make best use of the aestheticisation of their 
current lifeworlds. 
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Notes 


' All participants were both given written and verbal information about the project according to national 
ethical standards and legal protocols. They were given time to read the research protocol that covered relevant 
practical, ethical and legal aspects of the project as well as relevant documents, e.g. documents concerning 
written consent. They indicated ifthe data could be published in their original form or as blurred images. 
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? Participant observation always involves ethical considerations about how the participants’ situation influ- 
ences the data collected. This methodological bias cannot be avoided but was addressed in preparatory 
conversations with the participants. They all revealed that video-recording is acommon phenomenon and 
that they felt a bit awkward in the very beginning, but soon ignored the camera and the observer. We are 
aware that the presence of the cameras and the observer can have a significant influence on the participants, 
hence we addressed it directly and were sensitive to this issue in the analysis and interpretation of the data. 
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Cross-cultural Aesthetics: Aesthetic Contextualism 
and Ingroup Bias 


KOHINOOR M. DARDA‘" & ANJAN CHATTERJEE! 


Abstract: Almost every culture engages actively with the arts. Here, we explored how contextual 
information about content, artist, and technique impacts aesthetic experience. Contextual informa- 
tion increased aesthetic ratings for artworks in Indian and Northern American participants, espe- 
cially in people more open to experience, and Northern American participants with low art experi- 
ence. Contextual information reduced the ingroup bias for artworks, suggesting exposure to art may 
be a potential vehicle to mitigate prejudice against unfamiliar cultures. Similarities and differences 
in the routes to broader valuations of liking and beauty point to both anthropological universals and 
cultural specifics of the aesthetic experience. 


Keywords: art, culture, ingroup bias, aesthetics, context 


Introduction 


he arts are one of humanity’s most vital concerns, constituting an intrinsic part of how people 

live in the world. Most, if not all, cultures and societies engage actively with the arts — people 
sing, dance, design, narrate and listen to stories, and put on performances (Bamford, 2009). The arts 
reflect people and their social, cultural, and political dynamics and allow us to communicate beyond 
the boundaries of country and culture (Darda & Cross, 2022). Does this socio-cultural embedding 
influence how peoples across the world create and evaluate art? 

Philosophers have debated whether art has an inherent representational, expressive, or formal 
quality that makes it valuable independent of its cultural, social, moral or historical circumstances, or 
whether art is always embedded in its context which determines the interpretation of and engage- 
ment with art (Levinson, 2007). Research in the psychology of aesthetics and arts support aesthetic 
contextualism — aesthetic engagement of the viewer is influenced by contextual factors such as where 
one engages with an artwork (e.g., online or in a museum), the artwork’s title, or the sociocultural 
content depicted in the artwork (Pelowski et al., 2017; Belke et al., 2010; Darda & Cross, 2022; 
Darda, Christensen, & Chatterjee, 2023). Beyond the artwork, several contextual factors associated 
with the viewer (also referred to as individual differences) influence aesthetic experience. These 
include personality traits such as how open people are to experiences, their background, culture, 
memories or associations, and their art experience, expertise, and knowledge (Pelowski et al., 2017; 
Bullot & Reber, 2013). 

Previous work supporting aesthetic contextualism, however, had a western focus, with studies 
exploring primarily American/European art in Northern American and Western European popu- 
lations, with limited focus on other cultures (Che et al., 2018). Such an approach limits our knowl- 
edge and understanding of a universal perspective of aesthetic experience (Matsumoto & Juang, 
2003). With growing acknowledgement that almost 80% of research in psychology more broadly is 
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done on only 20% of the world’s population (Henrich, 2010), recent studies in empirical aesthetics 
have begun to focus on art and viewers from different cultures, and how culture might interact to 
generate an aesthetic experience for different populations (e.g., Bao et al., 2016; Yang et al., 2019; 
Darda et al., 2023; Darda & Cross, 2022; Darda & Cross, 2023; Darda & Chatterjee, 2023; Trawinski 
et al., 2021). 

Previous work in the field suggests a preference for art from one’s own culture compared to 
another (Bao et al., 2016; Darda et al., 2023; but see Darda & Cross, 2022). One explanation for such 
a preference is the process of enculturation. For example, research on music suggests that listeners 
internalise the characteristics distinguishing musical styles and cultures, and immersion within their 
sonic surrounds shapes their musical preferences (Campbell, 2011; Pearce, 2018). An alternative but 
not necessarily contrasting explanation is that of group bias (typically in-group favouritism and out- 
group dislike). In day-to-day lives, people show in-group biases by expressing favouritism for 
members of their own race, culture, sex, or ethnicity (e.g., Bernstein et al., 2007; Kubota et al., 2012; 
Rudman & Goodwin, 2004). Similarly, in the current context, an ingroup bias refers to a phenom- 
enon wherein individuals show a preference for art from their own culture compared to another 
(Bao et al., 2016; Darda et al., 2023; but see Darda & Cross, 2022). Indeed, it is possible that 
enculturation or cultural experience underlies this group bias. It is also possible that the ingroup 
preference might manifest because of a sense of cultural closeness or belonging with the artist or 
artwork, and/or viewers might use cultural identity as an art appreciation heuristic to make aesthetic 
judgements (Mastandrea & Umilta, 2021; Darda, Christensen, & Chatterjee, 2023). 

How malleable is this ingroup bias for art? Studies show that individuals with higher art experi- 
ence (as measured by the art experience questionnaire (AEQ), Chatterjee et al., 2010) tend to have 
a lower (or negligible) ingroup bias for artworks compared to those with lower art experience, 
suggesting art experience might play a role in reducing our ingroup biases (Darda & Cross, 2022; 
Darda, Christensen, & Chatterjee, 2023). A recent study from our lab found that contextual infor- 
mation by way of semantic text associated with an artwork about its content, the artist, or the artist’s 
technique influenced the aesthetic experience of Northern American participants, especially those 
who had lower art experience and were more open to experience (Darda & Chatterjee, 2023). 
Participants’ ratings of liking and beauty were higher, and ratings of complexity were lower when 
contextual information about an artwork was provided compared to when no information was 
provided. This contextual information also reduced the ingroup bias — Northern American partici- 
pants’ ratings of liking for Indian paintings (compared to European/American paintings) increased 
after reading contextual information about the paintings (compared to when no information was 
provided about the paintings). 

While these studies point toward the existence of an ingroup bias and its modulation by art 
experience or contextual information, the effect sizes reported are generally small. One reason for 
this could be the multifaceted nature of aesthetic ratings. In most studies, participants are asked to 
rate artworks ona Likert scale of 1 to 5 on how much they like the artwork, or how interesting it is. 
Yet liking something because one feels a sense of cultural closeness with the artwork or liking 
something because one finds the artwork exotic are different processes that may not be reflected 
when participants are asked to only rate on how much they like a painting. Thus, different routes to 
broader valuations such as liking might exist and vary across cultures for artworks belonging to 
different cultures. To explore how these routes are similar or different, we included ‘impact-on- 
viewer’ terms derived from Christensen et al. (2023) that might predict broader valuations such as 
liking. These 11 impact terms reflect the cognitive and affective effects artworks can have on 
viewers (what the artwork makes you think or feel) - angry, calm, compassionate, challenged, 
edified, enlightened, enraptured, inspired, interested, pleasure, and upset. 

Our goal in the current study is three-fold: 
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(1) To extend previous work on the influence of context in a non-western population, by 
exploring whether contextual information about the content, artist, and technique, and viewers’ 
art experience and openness to experience influence aesthetic ratings for Indian participants, 


(2) To explore whether contextual information modulates the ingroup bias by influencing 
aesthetic ratings for artworks belonging to another culture compared to one’s own, and 


(3) To investigate whether Indian and Northern American populations have different routes 
to broader valuations such as liking and beauty when viewing Indian and European/Ameri- 
can artworks. 


We used data from Northern American participants from a previous study (Darda & Chatterjee, 
2023) and combined it with new data from Indian participants in this study. We hypothesize that 
ingroup bias is contextually modulated and predict that context will reduce ingroup aesthetic ratings in 
Indian and Northern American participants. We further hypothesize that there are many routes to 
liking and predict that Indian and Northern American participants’ ratings of liking might be affected 
by different factors for Indian and European/American artworks. More specifically, we predict: 


(1) 1a. Like Northern American participants (Darda & Chatterjee, 2023), contextual infor- 
mation about the content, artist, and technique will influence aesthetic ratings for Indian 
participants such that contextual information (compared to no information) will enhance 
liking and beauty and decrease ratings of complexity in Indian participants. 1b. The effect of 
contextual information will be higher in participants with lower art experience and higher 
openness to experience. 


(2) 2a. Indian and Northern American participants will show an ingroup bias such that Indian 
participants will like and find more beautiful Indian artworks compared to European/American 
artworks, and Northern American participants will like and find more beautiful European/ 
American artworks compared to Indian artworks. 

2b. The ingroup bias will be modulated by art experience such that those with less art expe- 
rience will show a higher ingroup bias. 

2c. The ingroup bias will be further modulated by contextual information such that the 
ingroup bias will be lower when contextual information is provided about an artwork com- 
pared to when no information is provided. 


(3) Finally, aesthetic impact ratings (as derived from Christensen et al., 2023) will predict 
ratings of liking differently for Indian and European/American artworks for Indian and 
Northern American participants. 


Method 
Open Science Statement 


We report how the sample size was determined, all data exclusions, and all measures used in the 
study (Simmons et al., 2011; 2012). Data pre-processing, statistical analyses, and data visualisations 
were performed using R (v 4.1.2, R Core Team, 2018). Data analyses were preregistered on the 
Open Science Framework (https://osf-io/w8ck2). Mixed effects model analyses were executed using 
the /me4 package (v.1.1-28). Post-hoc tests were executed using the emmeans package (v.1.7.2). We 
used an alpha of 0.05 to make inferences and controlled for multiple comparisons using Tukey-HSD. 


Data Availability Statement 


Following open science initiatives (Munafo et al., 2017), all raw data and stimuli are available 
online for other researchers to pursue alternative questions of interest (https://osf.io/qj65x/). 
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Stimuli generation 


Stimuli included 16 images of representational artworks by various Indian and European/Ameri- 
can artists (see supplementary material for details). The 16 artworks were drawn froma larger set of 
36 artworks used by our lab in a previous study (Darda et al., 2023; Darda & Chatterjee, 2023). The 
36 artworks were normed on ratings of motion, balance, saturation, warmth, depth, and complexity 
ona Likert scale from 1 (low) to 7 (high). We divided these 36 artworks into four groups (for more 
details, see Darda & Chatterjee, 2023) with similar ratings of motion, balance, saturation, warmth, 
depth, and complexity. We further chose 4 artworks from each subset that included two artworks by 
Indian and two artworks by European/American artists across a variety of artistic styles and content. 
Thus, the four subsets of artworks (4 artworks per subset, 2 Indian, 2 European/American) used did 
not differ significantly in mean ratings of motion, balance, saturation, warmth, depth, and complex- 
ity (Table 1). Artworks from subset 1 were not preceded by any contextual information. Each 
artwork from subset 2, 3, and 4 was preceded by information about the content, artist, and technique 
respectively (see Tasks and procedure below for more details). 


Ratings Subset 1 (N=4) Subset 2 (N=4) Subset 3 (N=4) Subset 4 (N=4) 
Mean (SD) Mean (SD) Mean (SD) Mean (SD) 
Motion 5.11 [0.26] 5.00 [0.39] 5.06 [0.42] 4.98 [0.52] 
Depth 5.27 [0.18] 5.26 [0.09] 5.32 [0.11] 5.21 [0.13] 
Saturation 5.09 [0.35] 5.01 [0.23] 5.27 [0.27] 5.11 [0.22] 
Complexity 5.33 [0.14] 5.07 [0.24] 5.30 [0.14] 5.15 [0.08] 
Warmth 5.20 [0.27] 5.11 [0.17] 5.26 [0.23] 5.17 [0.28] 
Balance 5.29 [0.20] 5.33 [0.16] 5.33 [0.12] 5.27 [0.26] 
Contextual information No information Information intention lcamation 
about the content about the artist about the technique 


Table 1. Mean ratings of motion, depth, saturation, warmth, balance, and complexity across the three 
subsets of paintings. N = number of paintings, SD = standard deviation. 


Sample size justification 


An apriori power analysis (details in Darda & Chatterjee, 2023) suggested that with N=200 people, 
we had more than 80% power to detect an effect of contextual information (with four levels: no 
information, content information, artist information, technique information). We were able to 
recruit N=198 Northern American participants and N=125 Indian participants. With Northern 
American and Indian participants combined, we have sufficient power to detect the main effect of 
contextual information, but not the interaction effects. Therefore, findings from the interaction 
terms are suggestive, and not confirmatory. 


Participants 


Participants of Northern American origin (and residing in Northern America) were recruited on 
Amazon Mechanical Turk (MTurk), participants of Indian origin (and residing in India) were 
recruited by advertising on social media. Four hundred and thirty-six American participants and 
361 Indian participants started the experiment, and 380 American and 334 Indian participants 
completed it. As pre-registered, participants were excluded if they did not pass our attention checks 
(N=182 Americans, N=201 Indians, see the Tasks and Procedure section below for details on the 
attention checks), or were 2 standard deviations above or below the mean time taken to complete 
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the experiment (N=12 Americans, N=8 Indians). The high number of exclusions is due to the online 
nature of this study and the strict attention check questions we had to ensure good data quality. The 
final sample of participants included 198 American participants (96 men, 98 women, | non-binary; 
Mean, = 39:41, SD._= 11 .28) and 125 Indian participants (51 men, 69 women, 2 non-binary; 
Mean. ge = 24.60, SD, = 5. 97). Table S1 reports all participant demographics. Participants provided 


informed consent, and all study procedures were approved by the University of Pennsylvania IRB. 
All research was conducted in accordance with the Declaration of Helsinki. 


Tasks and procedure 


Participants completed a rating task, followed by questionnaires that assessed their art experience 
and openness to experience, with demographic questions at the end. Art experience was assessed 
using the Art Experience Questionnaire (AEQ; Chatterjee et al., 2010), and openness to experience 
was assessed using an Openness to Experience scale (OE) used in a previous study from our lab 
(Darda & Chatterjee, 2023). The 38-item openness to experience scale was derived from a total of 
68 items based on the NEO-PI-3 (Costa & McCrae, 2010) and the Big Five Aspects Scale (BFAS; 
DeYoung et al., 2007; for more details, see Darda & Chatterjee, 2023). 

In the rating task, participants viewed a total of 16 images of representational artworks by Indian 
and European/American artists, but were not explicitly told whether artworks were by Indian or 
European/American artists. Participants rated each artwork on the following variables: 


Liking; how much do you like this painting? [1=do not like at all, 5=like it very much] 
Beauty; how beautiful do you find the painting? [1=not at all beautiful, 5=very beautiful] 
Complexity; how complex do you think the painting is? [1=very simple, 5=very complex] 


Participants also rated each artwork on 11 ‘impact on viewer’ dimensions on a Likert scale from 1 
(not at all) to 5 (a great deal) derived from a taxonomy describing qualities of artworks, and the 
cognitive and affective effects artworks can have on viewers (Christensen et al., 2022). These eleven 
impact terms were preceded by the statement ‘this artwork made me think or feel...’ and included: 
angry, calm, compassionate, challenged, edified, enlightened, enraptured, interested, inspired, pleasure, and 
upset. The order in which ratings were presented was randomized across participants. As each 
artwork was rated ona total of 14 dimensions, we used only 16 artworks in the experiment to avoid 
fatigue effects in our participants. As we found similar results for our main dependent variables 
(liking, beauty, complexity) and the aesthetic impact terms, we present results from only the main 
dependent variables for our pre-registered linear mixed effects models in the main paper. The results 
for the aesthetic impact terms can be found in the supplementary material. 

The rating task was divided into four blocks. No contextual information was presented before any 
artworks from subset 1. Each artwork from subset 2 was preceded by information about the content 
of the artwork, each artwork from subset 3 was preceded by information about the artist, and each 
artwork from subset 4 was preceded by information about the technique used by the artist. The 
order in which these blocks were presented was randomized across participants. Content informa- 
tion was either descriptive or elaborative (one Indian and one Anglo European artwork was pre- 
ceded by descriptive content information, and the other Indian and Anglo European artwork was 
preceded by elaborative content information). Descriptive information included describing objects 
or colours or low-level features in the artwork, whereas elaborative information expanded more on 
what the artwork depicted (see Box 1). Each piece of information was followed by an attention 
check question to ensure participants were paying attention to and reading the information pre- 
sented to them before rating the artwork. Participants who had less than 90% accuracy on the 
attention check questions were excluded from the analyses. The entire experiment took around 30 
minutes for most participants (Mean,,,_,., = 31.50, SD 4. o, = 18.56), and participants were paid $4 
(Northern American participants) or Rs. 350 (Indian participants) as compensation. 
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Example Artwork Type of Contextual Information 


An artwork from subset 1 


No contextual information provided 


An artwork from subset 2 
with ‘descriptive’ content 
information 


This painting by John Sloan shows the interiors of McSorley’s bar, one of New 
York’s oldest bars, with its clientele standing at the bar. It depicts several working- 
class customers drinking around a wooden bar along with the bartender. The 
background is lined with paintings and other relics and objects. Dark and muted 
tones dominate the painting, brightened only by flesh tones, highlights of white and 
yellow, and dabs of orange and red. 
Attention check: McSorley’s bar is in which city? (Options: New York, Philadel- 
An artwork from subset 2 phia, Boston, Washington DC) 

with ‘elaborative’ content 

information 


This painting portrays an allegorical meeting between the artist with his patron. It 
is an interpretation of two mutually interested and interdependent characters who 
represent two different social classes and two different but interrelated roles in 
society. The patron’s status is portrayed by his demeanour and manservant. The 
artist on the right, is powerfully erect, with his head held high. The artist, whose 
role it is to wander and have no settled place in society is presented as equal to the 
man of wealth and social position. 

Attention check: Who is the meeting with? (Options: the artist and the patron, two 
An artwork from subset 3 random strangers, the dog and its manservant, the artist and his muse) 

with artist information 


This painting is by artist S. Elayaraja, whose paintings became world renowned for 
his realistic depiction of Tamilian women, their culture, tradition, and lifestyle. 
Born in 1979 in a small village in the South of India, Elayaraja was the youngest 
of eleven children. He drew inspiration from his experiences in a large family and 
made it part of his identity. He obtained Bachelor and Master of Fine Arts degrees 
from Kumbakonam and Chennai respectively, specializing in oil paintings, water 
colours, knife painting, and print making and photography. 

Attention check: What is the name of the artist you just read about? (Options: S 
An artwork from subset 4. Elayaraja, SS Rajamouli, Raja Ravi Verma, Swathi Thirunal) 

with technique information 


Louis Janmot’s paintings are a transition between romanticism and symbolism 
artistic styles, and the flawless finish is combined with a sense of mysticism. He 
had a preference for symmetry and repetition in his paintings and had a lot in 
common with pre-Raphaelite paintings in terms of content, colour, design, and 
emphasis on flowers and nature. He applied a design of well-defined contours, 
simple and dry colours, and a realism in presentation to his paintings. 

Attention check: Louis Janmot’s paintings have a sense of ——— in them. (Op- 
tions: mysticism, abstract expressionism, cubism, nihilism) 


Box 1. Example artwork and types of contextual information associated with it. All images used in 
this box are free from copyright restrictions. Image is taken from Darda & Chatterjee, 2023. 
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Data analysis 


In Darda & Chatterjee (2023), we found that contextual information influenced aesthetic engage- 
ment of representational artworks in Northern American participants. This aesthetic engagement 
manifested as changes in ratings of liking, beauty, and complexity, but only in participants with less 
art experience and those who were more open to experience. Therefore, as preregistered, in this 
experiment, first, we aimed to replicate these findings in Indian participants (H1a, H1b). 

For Indian participants only, for each of our main dependent variables (ratings of liking, beauty, 
and complexity), we ran two linear mixed effects models — one with the interaction between art 
experience and contextual information as a fixed effect (artexp model), and another with the interac- 
tion between openness to experience and contextual information as a fixed effect (openexp model) 
with by-subject and by-item random effects. In previous work, when we included both art experi- 
ence and openness to experience in the same model, we did not find meaningful differences. There- 
fore, we chose to pre-register and run separate models (instead of including a three-way interaction 
of openness to experience, contextual information, and art experience) to keep our model structure 
simple and allow us more power to detect our effects of interest. 

Both art experience and openness to experience were added as categorical variables. To do this, 
we centered AEQ and OE scores. Participants with centered AEQ scores above 0 were categorized 
as ‘high art experience’ and those with centered AEQ scores below 0 were categorized as ‘low art 
experience’ participants. Similarly, participants with centered OE scores above 0 were categorized 
as ‘high openness’ and those with centered OE scores below 0 were categorized as ‘low openness.’ 
Results were similar both when art experience and openness to experience were used as categorical 
variable (high experience and low experience; high openness and low openness) or when added to 
the model as a continuous variable (centered AEQ or OE scores). Art experience and openness to 
experience were coded with ‘high art experience’ and ‘high openness’ coded as 0.5 and ‘low art 
experience’ and ‘low openness’ coded as -0.5. 

The categorical variable of contextual information was coded using a simple coding style where 
every other level is compared to the reference level. No contextual information was used as the 
reference level, and each of the other levels (content information, artist information, technique 
information) were compared to the reference level separately. To control for effects of demographic 
variables, art experience, or openness to experience (OE), we further added age, education, total 
AEQ score (for the openexp model), and total OE score (for the artexp model) as fixed effects to the 
model. All continuous variables were centered to the mean by subtracting the mean from every 
value of the variable. 

The final models used were: 


Model artexp <- rating ~ 1 + contextual information" art experience + age + education + openness 
to experience + 1/sid + 1/itemno 

Model openexp <- rating ~ 1 + contextual information” openness to experience + age + education 
+ art experience + 1/sid + 1/itemno 


As preregistered, to explore the ingroup bias (H2a), we ran a linear mixed effects model with the 
interaction between participant culture (Indian, Northern American) and artwork culture (Indian, 
American/European) as a fixed effect and by-subject and by-item random effects. The categorical 
variables of participant culture and artwork culture were coded as 0.5 for Indian participants and 
artworks of Indian origin, and -0.5 for Northern American participants and artworks of American/ 
European origin. 


Model ingroup <- Rating ~ 1 + artwork culture*participant culture + age + education + art 
experience + openness to experience + 1/sid + 1/item 
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Next, to probe the influence of art experience on the ingroup bias (H2b), we added art experience 
to the ingroup model: 


Model ingroup_artexp <- Rating ~ 1 + artwork culture*participant culture*art experience + 
age + education + openness to experience + 1/sid + 1/item 


Finally, to probe whether contextual information influences the ingroup bias (H2c), we ran the 
following linear mixed effects model separately for participants with lower art experience and par- 
ticipants with higher art experience: 


Model ingroup_context <- Rating ~ 1 + artwork culture*participant culture*contextual infor- 
mation + age + education + openness to experience + 1/sid + 1/item 


Contextual information for the ingroup_context model was coded as -0.5 for the ‘no information’ 
condition and 0.5 for all other types of contextual information combined. 

Finally, as preregistered as an exploratory analysis (H3), to explore different routes to broader 
valuations such as beauty and liking, we had preregistered models separately for Indian and North- 
ern American participants for Indian and European/American artworks. However, we decided to 
run a more stringent test to explore different routes to broader valuations that would allow us to 
compare directly between Indian and European/American artworks separately for Indian and Ameri- 
can participants by including the interaction between each impact term and artwork culture. There- 
fore, we ran the following models in the current paper: 


Model Indian_routes (for Indian participants only) <- beauty/liking ~ 1 + angry*artwork 
culture + calm*artwork culture + compassionate*artwork culture + challenging*artwork cul- 
ture + edified*artwork culture + enraptured*artwork culture + enlightened*artwork culture 
+ inspired*artwork culture + interested*artwork culture + pleasure*artwork culture + 
upset*artwork culture + 1/sid + 1/itemno 

Model American_routes (for American participants only) <- beauty/liking ~ 1 + 
angry*artwork culture + calm*artwork culture + compassionate*artwork culture + 
challenging*artwork culture + edified*artwork culture + enraptured*artwork culture + 
enlightened*artwork culture + inspired*artwork culture + interested*artwork culture + 
pleasure*artwork culture + upset“artwork culture + 1/sid + 1/itemno 


For completeness, we report these models in the main paper and the preregistered models in the 
supplementary materials. 
Results 


The impact of contextual information (and its modulation by art experience and openness to experience) 
in Indian participants. 


For the art_experience model, results showed that the interaction between art experience and dif- 
ferent types of information did not predict ratings of liking, beauty, and complexity (all ps>0.10) 
except for the interaction between technique information and art experience that marginally pre- 
dicted ratings of beauty (8 =-0.21, p=0.075). Content information predicted ratings of complexity 
(B = -0.33, p=0.015), with higher ratings of complexity when no information was presented com- 
pared to when content information was presented. Artist information predicted ratings of liking, 
and marginally predicted ratings of beauty and complexity, with higher ratings of beauty and liking 
and lower ratings of complexity when artist information was presented compared to when no 
information was presented (liking: B = 0.50, p=0.011; beauty: B = 0.39, p=0.054; complexity: B = - 
0.24, p=0.075). Openness to experience predicted ratings of complexity, with higher openness pre- 
dicting higher ratings of complexity (6= 0.21, p=0.004) and marginally predicted ratings of beauty, 
with higher openness predicting higher ratings of beauty (6 = 0.10, p=0.068). No other main effects 
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or interactions were significant (see Supplementary Table 2). None of the contrasts were significant 
in post-hoc tests when correcting for multiple comparisons. The models explained 3.1% of the 
variance for liking, 3.9% for beauty, and 4.9% for complexity ratings. 
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Figure 1. The effect of contextual information and its modulation by art experience on liking ratings in Northern 
American and Indian participants. F or Indian participants, content predicted ratings of complexity, and artist informa- 
tion predicted ratings of liking; there was no modulation of art experience. For Northern American participants, 
information about content, technique, and artist predicted ratings of liking only in participants with low art experience. 
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Figure 2. The effect of contextual information and its modulation by openness to experience for liking ratings in 
Indian and American participants. Both Northern American and Indian participants with higher openness to experi- 
ence tended to like artworks more when artist information was presented. 
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For the openness_experience model, the interaction between content, artist, and technique informa- 
tion, and openness to experience significantly predicted ratings of liking and beauty (content: liking: 
B = 0.25, p=0.034; beauty: B = 0.35, p=0.001, artist: liking: B = 0.38, p=0.002; beauty: B = 0.39, 
p<.001, technique: liking: 6 = 0.25, p=0.033; beauty: 6 = 0.40, p<.001) and the interaction between 
openness to experience, and artist and technique information marginally predicted ratings of com- 
plexity (artist: complexity: 6 = 0.22, p=0.061, technique: complexity: B = 0.22, p = 0.071). Openness 
to experience positively predicted ratings of complexity (8 = 0.33, p = 0.039). Content information 
compared to no information predicted ratings of complexity (8 = -0.36, p = 0.006), artist informa- 
tion predicted ratings of liking and marginally predicted ratings of beauty and complexity (liking: B 
= 0.42, p = 0.029, beauty: B = 0.35, p = 0.080, complexity: 6 = -0.24, p = 0.071). No other main effects 
or interactions were significant (see Table S3). The models explained 3.7% of the variance for 
liking, 4.6% for beauty, and 6.3% for complexity ratings. 

Post-hoc tests suggested (after correcting for multiple comparisons) a marginal effect such that 
participants liked artworks more when artist information was presented to them compared to no 
information, but only in participants with high openness to experience (estimate = -0.61, SE = 0.22, 
95% CI [-1.23, 0.01], p = 0.055). No other contrasts or comparisons were significant when correct- 
ing for multiple comparisons in post-hoc tests. 


Ingroup bias in aesthetic appreciation, and its modulation by art experience. 


For the ingroup model, the interaction between participant culture and artwork culture signifi- 
cantly predicted ratings of liking and beauty, but not complexity (liking: B = 0.30, p<.001, beauty: B 
= 0.30, p<.001, complexity: B = 0.06, p = 0.205). Art experience positively predicted liking, beauty, 
and complexity ratings (liking: B = 0.46, p<.001, beauty: B = 0.41, p<.001, complexity: B = 0.29, 
p<.001). Openness to experience positively predicted ratings of complexity (complexity: B = 0.11, p 
= 0.026) and education marginally positively predicted ratings of beauty (beauty: B = 0.07, p<.070). 
The main effect of participant culture predicted ratings of liking and beauty (liking: B = 0.33, 
p=.001, beauty: B = 0.37, p<.001). No other main effects or interactions were significant (see Table 
S4). The models explained 14.4% of the variance for liking, 12.9% for beauty, and 6.0% for com- 
plexity ratings. 

Post-hoc tests suggested that overall, Indian participants had higher liking and beauty ratings 
compared to American participants (liking: estimate = -0.33, SE=0.10, 95% CI [-0.535, -0.13], 
p=0.001, beauty: estimate = -0.37, SE = 0.10, 95% CI [-0.56, -0.175], p<.001). Post-hoc tests for the 
interaction suggested Indian participants rated Indian artworks higher on liking and beauty com- 
pared to American participants (liking: estimate = -0.48, SE = 0.11, 95% CI [-0.75, -0.21], p<.001; 
beauty: estimate = -0.52, SE = 0.10. 95% CI [-0.77, -0.26], p<.001). No other comparisons were 
significant when correcting for multiple comparisons. 

For the ingroup_artexp model, the three-way interaction of participant culture, artwork culture 
and art experience predicted ratings of liking, beauty, and complexity (liking: B = -0.23, p=0.030, 
beauty: B = -0.28, p=.006, complexity: B = -0.25, p=.014). The two way interaction of artwork 
culture and participant culture predicted ratings of liking and beauty (liking: p = 0.34, p<.001, 
beauty: B = 0.30, p<.001, complexity: B = 0.07, p=0.163), the two way interaction of participant 
culture and art experience predicted ratings of liking, beauty, and complexity (liking: B = -1.45, 
p<.001, beauty: B = -1.27, p<.001, complexity: B = -0.76, p<.001), and the two way interaction of 
artwork culture and art experience predicted ratings liking and complexity, and marginally pre- 
dicted ratings of beauty (liking: B = 0.21, p<.001, beauty: B = 0.09, p=.066, complexity: B = 0.12, 
p=.020). The main effect of art experience positively predicted ratings of liking, beauty, and com- 
plexity (liking: B = 0.64, p<.001, beauty: B = 0.58, p<.001, complexity: B = 0.40, p<.001). The models 
explained 19.0% of the variance for liking, 17.1% for beauty, and 6.6% for complexity ratings (see 
Table $5). 
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Figure 3. The ingroup bias in Indian and Northern American participants. Indian participants liked Indian 
artworks more than European/American artworks. 


Post-hoc tests suggested that for liking ratings, American participants (but only those with lower 
art experience) liked European/American artworks more than Indian artworks (estimate = 0.31, SE 
= 0.12, 95% CI [0.08, 0.55], p=.008). Indian participants with higher art experience marginally 
liked Indian artworks more than European/American artworks (estimate = -0.23, SE = 0.13, 95% CI 
[-0.50, 0.03], p=.079). For beauty ratings, American participants with lower art experience found 
(marginally) European/American artworks more beautiful than Indian artworks (estimate — 0.23, 
SE = 0.12, 95% CI [-0.02, 0.44], p = .073), and Indian participants with lower art experience found 
(marginally) Indian artworks more beautiful than European/American artworks (estimate = -0.23, 
SE = 0.12, 95 CI [-0.46, 0.003], p=.053). For complexity ratings, only American participants with 
lower art experience rated European/American artworks higher on complexity than Indian art- 
works (estimate = 0.25, SE = 0.086, 95% CI [0.08, 0.42], p=.003). No other comparisons were 
statistically significant. 
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Figure 4. The modulation of the ingroup bias by art experience. Northern American participants with 
lower art experience showed an ingroup bias. 
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The impact of contextual information on the ingroup bias 


We preregistered separate models for participants with high and low art experience. However, as 
we did not find many differences in both groups (and we did not have power to perform a four-way 
interaction to statistically compare between participants with high and low experience), we decided 
to run the ingroup_context for all participants. Results for the separate models are reported in the 
supplementary material. 

For the ingroup_context model, the interaction between participant culture, artwork culture, and 
contextual information predicted ratings of beauty and liking but not complexity (liking: B = -0.45, 
p<-.001; beauty: B = -0.23, p=.033; complexity: B = 0.02, p=.886). The two-way interaction between 
participant culture and contextual information predicted ratings of complexity (complexity: B = - 
0.12, p=.023). The two-way interaction between artwork culture and participant culture continued 
to predict beauty and liking ratings (liking: B = 0.41, p<.001; beauty: B = 0.35, p<.001). The main 
effect of art experience predicted ratings of liking, beauty, and complexity (liking: B = 0.46, p<.001; 
beauty: B = 0.41, p<.001; complexity: B = 0.29, p<.001), openness to experience predicted complexity 
(complexity: B = 0.11, p=.026), contextual information predicted liking and complexity ratings 
(liking: B = 0.24, p=.033; complexity: B = -0.16, p=.040), and participant culture predicted liking and 
beauty ratings (liking: B = 0.33, p<.001; beauty: B = 0.36, p<.001). None of the other main effects and 
interactions were significant (see Table S6). 
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Figure 5. The modulation of the ingroup bias by contextual information for Indian and American participants. Indian 
participants showed higher ratings of liking after contextual information was presented for American/European 
artworks but not for Indian artworks. Northern American participants showed higher ratings of liking after contextual 
information was presented for Indian artworks but not for American/European artworks (although this was signifi- 
cant at our statistical threshold). 
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For the three-way interaction, post-hoc tests suggested that for Indian participants, contextual 
information increased ratings of liking and beauty for European/American artworks (liking: esti- 
mate = -0.50, SE = 0.17, 95% CI [-0.82, -0.17], p=.003; beauty: estimate = -0.39, SE = 0.17, 95% CI 
[-0.73, -0.05], p=.024) but not for Indian artworks. No other contrasts or comparisons were statis- 
tically significant. Figure 5 shows that for both Indian and American participants, we see that ratings 
of liking for artworks of the other culture were higher when contextual information was presented 
(although this was not significant at our statistical threshold for American participants). 


Different routes to broader valuations of liking and beauty 


For the Indian_val model, the interactions between angry ratings and artwork culture, edified 
ratings and artwork culture, and enlightened ratings and artwork culture marginally predicted 
liking ratings for Indian participants (angry: B = -0.06, p=.095; edified: B = -0.08, p=.077; enlight- 
ened: B = 0.10, p=.054) and the interaction between upset ratings and artwork culture predicted 
ratings of liking for Indian participants (upset: B = -0.12, p=.036). How calm (p<.001), compassion- 
ate (p=.002), enraptured (p<.001), inspired (p<.001), interested (p<.001), and how much pleasure 
(p<.001) Indian participants felt when viewing the artwork positively predicted ratings of all art- 
works. How angry Indian participants felt when viewing the artworks negatively marginally pre- 
dicted ratings of all artworks marginally (p=.053). Post-hoc tests suggested that lower ratings of how 
angry participants felt predicted higher ratings of liking for European/American artworks (p=.013), 
but not Indian paintings (p=.801). Higher ratings of how edified Indian participants felt when 
viewing the artworks predicted higher ratings of liking for European/American artworks (p=.035) 
but not Indian artworks (p=.852). Higher ratings of how enlightened Indian participants felt when 
viewing the artworks predicted higher ratings of liking for Indian artworks (p=.018) but not Euro- 
pean/American artworks (p=.778). Lower ratings for how upset participants felt marginally pre- 
dicted higher ratings of liking for Indian artworks (p=.054) but not European/American artworks 
(p=.357). 

a the American_val model, the interaction between how compassionate participants felt and 
artwork culture predicted ratings of liking for American participants ( = 0.08, p=.025). How calm 
(p<.001), compassionate (p<.001), enlightened (p<.001), enraptured (p<.001), inspired (p<.001), 
interested (p<.001), and how much pleasure (p<.001) American participants felt positively predicted 
ratings of how much participants liked all artworks overall. Post-hoc tests suggested that higher 
ratings of how much compassion American participants felt when viewing the artwork predicted 
higher ratings of liking but only for Indian artworks (p<.001) but not European/American artworks 
(p=.100). 

a beauty ratings, for the Indian_val model, the interaction between artwork culture and interest 
ratings predicted ratings of beauty for Indian participants (8 = 0.10, p=.024), and the interaction 
between artwork culture and compassionate ratings, and the interaction between artwork culture 
and upset ratings marginally predicted ratings of beauty for Indian participants (compassionate: B = 
-0.08, p=.093; upset: B = -0.10, p=.075). How calm (p<.001), challenged (p=.026), compassionate 
(p<.001), enraptured (p=.021), inspired (p=.001), interested (p<.001), and how much pleasure (p<.001) 
Indian participants felt predicted beauty ratings for all artworks. Post-hoc tests suggested that higher 
ratings of how much compassion Indian participants felt when viewing the artworks predicted 
higher ratings of how beautiful they found European/American artworks (p<.001) but not Indian 
artworks (p=.131). Contrasts for how upset participants felt were not statistically significant when 
comparing between Indian and European/American artworks. Higher ratings of how interested 
Indian participants were in the artwork predicted higher ratings of liking more strongly for Indian 
artworks (p<.001) compared to European/American (p<.001) artworks (p=.025). 

For beauty ratings, for the American_val model, the interaction between artwork culture and 
interest ratings, and the interaction between artwork culture and upset ratings predicted ratings of 
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beauty for American participants (interested: B = 0.14, p<.001; upset: B = 0.10, p=.016). How calm, 
compassionate, enlightened, enraptured, inspired, interested, and how much pleasure participants 
felt (all ps<.001) positively predicted ratings of beauty for American participants for all artworks. 
Post-hoc tests suggested that higher ratings of how interested participants were in the artwork more 
strongly predicted higher ratings of beauty for Indian paintings (p<.001) compared to European/ 
American paintings (p<.001) for American participants. Lower ratings of how upset American 
participants felt when viewing the artwork predicted higher ratings of how beautiful they found the 
artworks, but only for European/American artworks (p=.034) but not Indian artworks (p=.240). 
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Figure 6. Different routes for Indian and European/American artworks for Indian (left) and American (right) 
participants for liking ratings. The X-axis denotes beta estimates for the models (separate models for Indian and 
European/American paintings), and Y-axis shows the predictor variables. 
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Figure 7. Different routes for Indian and European/American artworks for Indian (left) and American (right) 
participants for beauty ratings. The X-axis denotes beta estimates for the models (separate models for Indian and 
European/ American paintings), and Y-axis shows the predictor variables. 
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General Discussion 


The current study aimed to explore how contextual factors associated with the artwork and the 
viewer might interact to generate an aesthetic experience. Combining data from Northern Ameri- 
can and Indian participants, and using European/American and Indian art, we explored whether 
contextual information such as content, artist, and technique information, and viewers’ art experi- 
ence and openness to experience influence aesthetic experience in Indian participants and Northern 
American participants (Darda & Chatterjee, 2023). We investigated the ingroup bias, and its modu- 
lation by art experience and contextual information. Finally, we explored whether routes to broader 
valuations of liking and beauty might differ for Indian and European/American artworks for Indian 
and Northern American participants. 

Our results suggest that contextual factors influenced both Northern American and Indian partici- 
pants, but the modulation of this effect by art experience and openness to experience differed across 
cultures. Contextual information reduced the ingroup bias (more so for Indian participants), and 
routes to broader valuations of liking and beauty showed some differences between Indian and 
European/American artworks for Indian and Northern American participants. Below, we evaluate 
each of these findings and our questions in more detail. 


The impact of contextual information (and its modulation by art experience and openness to experience) 
in Indian participants. 


In line with previous research (e.g., Leder et al., 2006; Swami, 2013) and similar to Northern 
American participants (Darda & Chatterjee, 2023), contextual information influenced aesthetic 
ratings such that ratings of liking and beauty were higher, and ratings of complexity were lower 
when artist-related contextual information was presented to Indian participants compared to when 
no information was presented. Similarly, content-related contextual information reduced ratings of 
complexity compared to when no information was presented to participants. Technique informa- 
tion did not impact aesthetic ratings. These findings are consistent with the fluency theory which 
suggests that ease of processing increases an artwork’s appreciation and suggests that contextual 
information promotes greater fluency (Reber et al., 2004). The possibility that artist- and content- 
related information enhanced fluency for the artworks is supported by the observation of decreased 
complexity ratings in those conditions. 

Investigations in empirical aesthetics have shown that individual variability, such as openness to 
experience and art experience (e.g., Leder et al., 2004; Fayn et al., 2015), also impacts aesthetic 
appreciation. Here, we find that Indian participants (like Northern American participants) also 
show an effect of contextual information modulated by openness to experience such that artist- 
related information influenced aesthetic ratings but only in participants with higher openness to 
experience. People with higher openness to experience seek novelty in artworks and therefore, 
(novel) information about the artwork may have a greater impact on their aesthetic experience and 
judgements (Fayn et al., 2015). 

However, the modulation of the effect of contextual information by art experience was not found 
in Indian participants. Indian participants with both higher and lower art experience were influ- 
enced similarly by contextual information. These findings contradict previous research that show 
differences in participants with high and low art experience. Leder et al. (2004) suggest that people 
with higher art experience and knowledge may view artworks differently to those with lower art 
experience, engaging more with the style of the artwork than the information associated with it. 
Thus, processing fluency may already be higher in those with higher art experience and therefore 
they may be less influenced by contextual information. 

One possible explanation for the discrepancy between Indians and Americans is the measure of art 
experience used. The AEQ (Chatterjee et al., 2010) and other measures used commonly in the field 
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to measure art knowledge or aesthetic sensitivity (Specker et al., 2018; Schlotz et al., 2020) were 
developed for and validated among western populations. Thus, any differences across cultures might 
represent the limits of these measures to generalise across cultures. The current findings are also in 
line with recent work in cross-cultural empirical aesthetics that also suggests differences in modula- 
tion by art experience across cultures (Darda & Cross, 2022; Darda, Christensen, & Chatterjee, 
2023). An important goal for future research would be to develop and/or validate measures that can 
be used outside of traditionally over-represented research samples in empirical aesthetics and psy- 
chology more broadly (Golbabaei et al., 2022). 


Ingroup bias in aesthetic appreciation, and its modulation by art experience and contextual information. 


In line with previous research, we found an ingroup bias - Indian participants with high and low 
art experience preferred Indian artworks more than European/American artworks, and Northern 
American participants with low art experience preferred European/American artworks more than 
Indian artworks (Bao et al., 2016; Yang et al., 2019; Darda et al., 2023). Similar to the modulation of 
art experience on the effect of contextual information, we did not find a modulation of art experi- 
ence on the ingroup bias in Indian participants. 

For Northern American participants, a modulation by art experience is perhaps explained by the 
uncertainty-identity hypothesis (Mastandrea et al., 2021). This hypothesis suggests that when par- 
ticipants are unsure about their own identity, they might resort to using national/cultural identity as 
an art appreciation heuristic. That is, if participants are unsure about art and are asked to give an 
opinion about it, they may resort to using group identification to resolve their uncertainty and use 
cultural identity as an art appreciation heuristic. A discrepancy in findings for Indian participants 
might suggest that Indian participants, whether with lower or higher art experience, continue to use 
cultural identity as an art appreciation heuristic. A more likely explanation however may be that as 
mentioned before, art experience as measured in the Northern American context may not be similar 
in an Indian context. For art experience to be measured across cultures, a crosscultural approach to 
art and aesthetics is crucial. For instance, broadly speaking “western” models of museums and cura- 
torial practices may differ significantly from “non-western” traditions and practices (Peers & Brown, 
2007; Kreps, 2006). Future tools and measures of art experience will have to consider these cross- 
cultural differences. 

Our results also suggested a modulation of the ingroup bias by contextual information. Indian 
participants showed a reduced ingroup bias when contextual information was presented compared 
to when no contextual information was presented. While we did not find statistically significant 
results for Northern American participants, there was a trend for a reduced ingroup bias in Northern 
American participants with lower art experience. In our previous study, however, we did find a 
statistically significant effect in Northern American participants, when Indian participants were not 
included in the model (Darda & Chatterjee, 2023). A non-significant effect in the current study, 
therefore, is more likely explained by the small size of the effect, and a lack of power to detect this 
effect (a three-way interaction) with our current sample size and number of items as opposed to the 
absence of an effect. 

An ingroup bias and its modulation by contextual information can be explained in line with the 
uncertainty-identity hypothesis. When uncertainty about making an opinion is reduced (by way of 
providing more information), participants might resort less to using cultural identity as an art appre- 
ciation heuristic. Alternatively, increased exposure to more information about unfamiliar cultures 
might increase aesthetic ratings toward artwork of that culture. While the current study did not tease 
apart the processes underlying the ingroup bias and its modulation, future work can explore whether 
preference for one’s own culture emerges from cultural closeness, social identity, enculturation, or 
increased familiarity or exposure to the cultural content of the artwork. Nonetheless, irrespective of 
the underlying processes, the current findings open the possibility of how one might be able to 
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influence preference for creations from one’s own culture compared to another, and whether this 
might apply to non-art contexts. 


Routes to broader valuations of liking and beauty 


Previous studies have used liking or beauty ratings to index art judgments. However, aesthetic 
experience is a multicomponent process that encompasses more than just aesthetic preference, in- 
cluding experiences of beauty, sublimity, and complex cognitive and emotional evaluations (Leder 
et al., 2004; Chatterjee & Vartanian, 2016). We used 11 impact terms that tapped into cognitive and 
affective evaluations, and predicted that different impacts would contribute to broader valuations of 
liking and beauty in Indian and Northern American participants for Indian and European/Ameri- 
can artworks. 

For example, even though liking ratings by Indian participants might be similar for Indian and 
European/American paintings, Indian participants might like Indian paintings because they feel a 
sense of cultural closeness whereas they might like European/American paintings because they are 
novel to them. Thus, different factors may contribute to their aesthetic experience even though 
ratings of how much participants like paintings from their own culture compared to another might 
be similar. We found both similarities and differences in the aesthetic impacts that predicted beauty 
and liking ratings. Similarities in which impacts contribute to liking or beauty ratings are expected 
given that some aspects of the aesthetic experience of artworks might be universal (Che et al., 2018; 
Darda & Cross, 2022). However, we also found some differences in how Indian and Northern 
American participants ratings of liking for Indian and European/American artworks were predicted 
by aesthetic impacts. Although these differences were small, they point toward differences in how 
artworks of different cultures might be evaluated cross-culturally. 

For instance, for Indian participants, lower ratings of how upset participants felt on viewing the 
painting predicted higher ratings of liking for Indian artworks, but not American/European art- 
works. On the other hand, higher ratings of how angry Indian participants felt predicted lower 
ratings of liking for American/European artworks but not Indian artworks. Upset is more semanti- 
cally similar to how uncomfortable or anxious people feel, while angry is similar to how threatened 
or offended people feel on viewing artworks (Christensen et al., 2022). People may not like to feel 
uncomfortable or upset by artworks that belong to their own culture which contributes to how 
much they like artworks from their own culture more than artworks from another culture. People’s 
liking may also be more strongly influenced by how much they feel threatened or offended or angry 
by artworks from another culture, but this may not be similar for artworks from their own culture. 
For American participants, higher compassion ratings predicted higher liking ratings for Indian 
artworks but not American/European artworks, suggesting that more compassion when viewing an 
artwork from another culture might lead to higher liking for that artwork, but this process may not 
be similar for artworks from one’s own culture. 

These findings suggest that although people across cultures might have a universal aesthetic expe- 
rience per se, the route that leads to the construction of an aesthetic experience might differ depend- 
ing on cultural context. Indeed, the aesthetic triad model (Chatterjee & Vartanian, 2014) suggests 
that aesthetic experience is brought about by an interaction of sensory-motor, emotion-valuation, 
and knowledge-meaning systems in the brain. An interesting direction for future research would be 
to investigate how low-level visual properties of an artwork such as symmetry, brightness or contrast 
might contribute to the aesthetic experience of Indian and European/American artworks along with 
characteristics of an artwork, what viewers think or feel about the artwork (Christensen et al., 2022), 
and how different brain systems are engaged similarly or differently across cultures in constructing 
the aesthetic experience. As they stand, our results point to both anthropological universals as well as 
cultural specifics of human art creation and art appreciation. Art is universal as it arises from brain 
systems that are common across cultures, but these neural systems are flexible and dependent on 
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contexts and experiences, thus making art appreciation variable and diverse across cultures (Nadal & 
Chatterjee, 2019). 


Implications 


The current findings have implications for the fields of arts, empirical aesthetics, neuroaesthetics, 
as well as psychology more broadly. Neurocognitive models of aesthetics suggest sensory inputs 
combine with our emotional responses, and contextualise within our cultural backgrounds, memo- 
ries, individual associations and past experiences to bring about an aesthetic experience (Chatterjee 
& Vartanian, 2014). Future research using neuroimaging techniques could investigate which sys- 
tems are engaged when processing contextual information. For instance, it is possible that artist 
information may engage the knowledge-meaning systems but have little impact on visual process- 
ing (sensory-motor system) directly. Alternatively, content-related information may influence vi- 
sual processing but have little impact on emotion-valuation systems. 

The current findings also have implications for museum or exhibition curation and arts educa- 
tion. Given our findings, it seems imperative to keep in mind the curatorial background of the 
museum or exhibitions, as well as the viewers to which it caters (Brieber et al., 2015; Darda & 
Chatterjee, 2023). Further, the possibility that contextual information can reduce ingroup bias in an 
art context opens possibilities for the mitigation of outgroup prejudice. As the world becomes more 
fractured because of social, political, economic, and geostrategic factors, it is imperative to identify, 
mitigate, and counter these biases and prejudices. The arts have long been promoted as one medium 
to help us develop empathy, theory-of-mind, prosocial behaviour, and impact attitudes and out- 
group prejudices (Kou et al., 2020; Dodell-Feder & Tamir, 2018; Mar & Oatley, 2008). An exciting 
avenue for future research would be to explore how and whether the consumption of art in context, 
especially unfamiliar art, can impact biases and prejudices in non-art contexts. This line of research 
is especially relevant and important in today’s times when funding cuts for the arts are on the rise 
(The Guardian, October 2022; Micallef, 2021; State Arts Agency Revenues, 2021). 


‘Penn Center for Neuroaesthetics, University of Pennsylvania, Philadelphia, USA 
A dvancement and Research in the Sciences and Arts (ARISA) Foundation, Pune, India 
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Art_ificial Intelligence: Dreams, Data, and Neuro- 
aesthetics in the Age of Al 


SADE J. ABIODUN & LUKE NICKEL 


Abstract: In an age where the worlds of art and science continuously collide, scholars and storytellers 
alike are increasingly using Artificial Intelligence (AI) to analyze and represent their internal worlds. 
As Al becomes further enmeshed in the everyday experience of humans and other sentiences across 
the planet, questions begin to arise concerning ethical representation of the selfand other, techno- 
logical anthropomorphism, and artistic expression. Our dialogue contains several modalities: text, 
poetry, artistic thought experiments, speculative compositions and images. Our aim is not to pro- 
vide answers to these ever-accelerating questions about Al, but to open up and expose the grooves of 
various narratives surrounding AI by engaging with experimentation, dialogue, and speculation as 
modes of discovery. 

Each section of the text begins with a poetic provocation by S. J. Abiodun, and continues on with 
an exchange between the two authors, fluidly leaping between our individual and collective knowl- 
edge as researchers, artists, and interdisciplinarians. 


Keywords: artificial intelligence, dreaming, neuroaesthetics, research-creation 


1. Introduction 


if androids do dream of electric sheep, 

are they able to paint them? 

are they able to add detail 

to the small tufts of wool 

and note the sounds of their virile bleating? 


are they able to describe and etch 
each blade of grass in the pasture 
over which their fence runs 

over which their sheep hop? 


or do they simply envision these creatures 
as components within an iterative system 
another metric of the goal-oriented process 
of lulling themselves into standby? 


C= day discourse around technology and society characterizes Artificial Intelligence (AI) as 
an unprecedented realm of computational advancement that may rival, if not supersede, hu- 
man cognitive abilities (Abiodun 2018). But AI, at its core, is not a new concept, and the invention 
of tools for creative mediation and intellectual exploration has been practiced since the first piece of 
flint was used to coax a flame, or the first paintbrush selected to graze a taut canvas. Given the 
potential for these emerging tools to challenge our perspective on the traditional dynamic between 
human expression, technology and art, we offer the following exchange as a means of exploring the 
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many intersections between neuroaesthetics, embodied artistic practice, and Artificial Intelligence. 
Our exchange moves through various ‘grooves’, guided by the poetic stanzas that preface the sec- 
tions, that take the reader and the authors through explorations of text and context (reflections on 
scientific and artistic applications of Al), exploration of inputs and outputs (creative experiments 
using Al intermediates), and discussion of the application of quantitative and qualitative frame- 
works in a multidimensional world. 

Our dialogue begins with an observation by S. J. Abiodun: in Fall of 2022, there was a notable 
uptick in social media users that were illustrating their dreams using artificial intelligence image- 
generation models. Many users spoke to the experience of being able to generate representations of 
their internal worlds, core identities, and imagined futures as liberating and euphoric, especially 
when they were given direct access to the creative potentials even without traditional training in 
programmatic or artistic disciplines (Keller 2022). As the conversation has evolved, the impact of 
these emerging technologies on an increasingly fragile labor market — where artists and creative 
practitioners are trying to adapt with the looming possibility of their process and value being 
outsourced — also raises major questions about the ethical nuances of automated creativity, process 
oriented vs. product oriented economies, and human-centric perspectives in a technocapitalist 
landscape. Together, these observations of the intertwining of creative, cognitive, and artificial 
worlds sparked a question: what are the implications of using generative AI technologies to mediate 
creative expression of our realities? Rather than answering the question directly, we instead propose 
to work through it using the tools at our disposal as interdisciplinarians: writing, researching, creat- 
ing art, and engaging in the speculative. Our aim is not to provide definitive answers to ever- 
accelerating questions about AI, but to highlight elements of the parallel narratives surrounding Al 
by engaging experimentation, dialogue, and speculation as modes of discovery. 

This dialogue features two core participants, as well as numerous nodes of input from their adja- 
cent communities. S. J. is a scientist and artist interested in emotion, subjective experience, and 
neurocinematics—the neuroscience of visual media. Luke Nickel is a multidisciplinary music com- 
poser and artist exploring memory, transmission, and roller coasters. The pair met at an interdisci- 
plinary summer institute that brought together scientists and ‘storytellers’—artists, science commu- 
nicators, and writers—to discuss various aspects of human, animal, social and artificial intelligences. 
Conversations related to this dialogue date back to November 2022 and continued through April 
2023. What began with an observation about AI and dreaming led to a larger discussion weaving 
together the subjects of expression, mediation, and narrative. 

Philip K. Dick’s 1968 novel, Do Androids Dream of Electric Sheep? contains an important duality: 
the androids to which the novella refers do not sleep or dream, whereas humans do. In the title of this 
piece, a classic reference in western pop culture to Al sentience and technological futures, Dick elides 
the notion of sleeping/dreaming with that of desire—desire/dreaming and sleeping/dreaming both 
being essentially human qualities that exist in opposition to those artificial qualities of the android. 

When we dream, we create and experience multimodal hallucinations that often contain frag- 
mented narratives. When we wake, we can apply these narratives to our life. We can analyze their 
component parts to investigate the past experiences to which they relate, or we can view them as 
predictive of possible futures. What happens when we invite artificial intelligence into this pipeline? 
What happens when we imagine these technologies as a dreaming partner, a co-dreamer that can 
reflect aspects of our dreams back to us through the probabilistic prism of highly biased human 
narratives—both images and text? When we use computational tools to interface with our dreams— 
whether through text prompt, linguistic analysis, image creation, or interpretation of electromag- 
netic impulses—we invite technology into the process of interfacing our inner worlds. 

We might also ask ourselves: who and what else do we invite into the dreaming pipeline in these 
scenarios? For example, we can envision the bones of past training data—both voluntarily and 
involuntarily provided—algorithms and code made by sources both benign and corporate, or 
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outsourced workers processing and censoring data as well as researchers in labs. We may imagine 
artificial intelligence as a co-dreamer, but it also encompasses a multitude of actors and influences, 
data and technologies, pasts embedded into presents and futures. 

The mediation between internal and external worlds is also discussed in the context of psychology 
and brain sciences, particularly in explorations of cognitive phenomena that elucidate how mental 
(internal) context interacts with, and is shaped by stimuli in our environmental context (Franklin 
2020, Bellana 2022, Brandman et al 2022). When folding dreams and imagination into the realm of 
internal world-building, it evokes a key question: what (or who) is the mediator and what (or who) 
is the medium? Many neuroscientists would consider the brain to be a computational medium 
through which input and output interface, and thus adopt the role of interpreter when using empiri- 
cal methods to analyze neuroimaging and behavioral data (Baria & Cross 2021). In this scenario, 
new AI technologies—such as neural network models of language (BERT, ChaptGPT) and imagery 
(Stable Diffusion)—take on a dual role as well, not only analyzing (and mediating) the data, but 
iterating on these inputs as ‘training’ to subsequently predict outcomes and generate new informa- 
tion (Horikawa 2013, Floridi 2020, Jain et al 2020, Dehouche & Dehouche 2023, Takagi & Nishimoto 
2023). When Al is considered as medium and scientists as mediators, does this map onto the usage in 
the wider world including creative practice? Or are we moving to new configurations when we 
request that these AJ agents not only extract, but also express? 


2. Al, Dreaming, Image-Making and Replacement 


when we think of androids dreaming of electric sheep 
&G&& instead 

task them with making sense of 

the sheep that we dream of 

the pastures that we roam in 

can our dreaming droid situate itself 

within the context and frame 

that stretches beyond 

the data, the outcome, the quantity of fin;out} put 
&G&& instead 

seek to behold 

the quality of Lim;ex}pression and interpretation 
of subjective subjects 

and contextually textured matter 

grounded and granular and gauged 


by human reality, human experience? 


Dreaming as Image-Making 


[Luke Nickel]: As an artist, I often try to understand artistic and cultural phenomena by partici- 
pating in them. Over the past number of years, I have explored AI in many aspects of my creative 
practice. For example, in early experiments I trained an image model on documentation of an 
interdisciplinary arts festival that I directed to eventually ‘curate’ an artificial festival of images as my 
final edition before leaving the post. In another project, I wrote text collaboratively with GPT-2 to 
create a ‘spell’ transforming audiences into roller coasters in an act of gravitational witchcraft, 
inspired by collaborative human-Al writing techniques described by K Allado-McDowell (“Inter- 
dependence”). Conducting these and other artistic experiments with AI has allowed me to feel some 
sense of control in a time when tools and technologies seem to be thrust upon artists in an ever- 
accelerating spiral. 

And so, when S. J. mentioned seeing twitter and reddit users illustrating their dreams using Midjourney, 
an AI text-to-image model that sprung to popularity in 2022, I decided to try to do the same. 
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Using tools such as Midjourney as an artist does not come without complications. 2022 was the 
year that a large and very public backlash against Al image-generation occurred: starting with Kim 
Jung Ji’s unexpected passing and posthumous use of his images to train an image model (Deck 2022), 
Jason M Allen’s state fair win with an Al-generated painting (Roose 2022), and leading to a growing 
outcry against the Lensa app and Laion dataset used to train Midjourney after private medical 
photos were found in the dataset (Xiang 2022). In my practice I normally would consider data 
provenance highly important, favoring my own datasets and trained models; however, in this ex- 
periment I wanted to use the most widely available tools possible. 

I began by journaling my dreams during the months of January and February 2023; however, 
generally my dreams in this period felt very short and fragmented, and my written recordings of 
them correspondingly brief. Despite some attempts at inputting them into image generation mod- 
els, both my writing and memory of the dreams lacked enough detail to create any satisfying images. 
IfI performed a similar experiment again, I might try to cultivate longer written recollections of the 
dreams themselves, as I believe that the brevity of my writing did not lend itself to prompt creation. 


...firepit sewage overflow on the beach mopping with towels... 
...what do you want your party to be like? “Not happening” nonbinary party doula... 
.. pissing ona chalk line that someone is painting... 


writings from my dream journal, Jan 2023 - Feb 2023 
g 'Y J 


Sacrificial replacement and the loss of multimodality 


I have a number of dreams that I can recall that contain memorable images—particularly those 
from when I was as young as three years old—but I worried that by using an AI to illustrate them, I 
would replace my original mental images with their Al-illustrated versions. These dreams and 
memories from such early ages are sparse, and thus feel precious. Of course, my memories of these 
dreams have already been replaced by my own recall from memory, and retelling to others. How- 
ever, I felt concerned about this technological replacement because rather than my own home- 
brewed displacements through writing or oral transmission, they could be replaced by images and 
algorithmic content from external sources—almost as when readers are anxious about having their 
own images of book characters replaced by those of a major motion picture adaptation. I did find a 
dream from a diary from 2017 that I was willing to ‘sacrifice’-—a short dream taking the form of an 
ambiguous fashion photoshoot. I tried to distill the dream’s images into coherent text prompts, 
which involved summarizing the dream’s overall situation and its most striking features and colors. 
Then, I input them into Midjourney, where my text would be transformed into Al-generated 
images (using Midjourney) as seen below. 
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Left: An enormous brutalist concrete diving platform indoor pool, a small diving male bearded model 
wearing a billowing silk yellow avantgarde garment issey miyake diving headfirst in motion, action 
shot, vogue photoshoot, wide angle shot, faraway, film photo 


Right: Indoor brutalist swimming pool 3 - level olympic concrete high - rise diving boards, a diving 
bearded male model mid - dive hands outstretched toward pool wearing billowing flowing yellow silk 


high - fashion garment issey miyake, film photo, vogue, glossy, hq 


The first thing I noticed is that it was remarkably challenging to create an image that accurately 
represented my recall of the dream. Dreams are multimodal, containing complex sense memories, 
fragmented narratives that ambiguously shift agency and perspective, and are layered with anach- 
ronistic feelings. Writing already loses some more sensory dimensions, and a flat, unmoving image 
loses even more. Dream scholars and analysts such as Hall and Van De Castle have created many 
categories for content analysis that attempt to capture these modalities, such as locational and emo- 
tional information (1966). Much of this information is lost when converting a dream to a text-based 
image prompt, in particular any ideas that point to the flow of time, changes in perspective, and 
abstraction. In the case of this dream-turned-prompt, the nature of a fashion photoshoot matched 
the vocabulary found within Midjourney’s data set quite clearly. Very quickly I had to abandon the 
idea of making a faithful interpretation to making a high quality image. 

Whereas my above experiments were nearly instantaneous, earlier work with AI took hours and 
even days to garner any results. In 2020-2021, the timescale of generating images using AI genu- 
inely felt like dreaming. Projects such as training an image model on all the open source images of 
roller coasters from wikipedia or generating images trained on single frames of videos could some- 
times take up to eight hours. I would half wake up at 5 am to move my mouse to keep my Google 
Colab environment running in order to continue generating images, often viewing images in a state 
of half-waking. The images I generated frequently surprised me with their dark and illogical glitchiness. 
The time-scale and complexity of the process felt tied to my surprise and delight in the results. 

Now the images I created in Midjourney do not elicit any surprise or curiosity in me. I will not 
proclaim that all Al-generated images are hollow or uninteresting, because that would discount 
excellent work in the field by artists such as Holly Herndon and Mat Dryhurst. But these quickly 
generated images made me feel more a part of a dystopian algorithmic future, summoned out of an 
endless scroll on deviantart, and returning to that void after my eyes leave them. 


Involuntary use of data 


While in my case my actions were voluntary and experimental, for many, participation in this 
algorithmic future— where the self, artwork and algorithm are fused—is involuntary. Multiple artists 
have attempted to file lawsuits against Midjourney, and at the same time there have been numerous 
attempts at creating tools that render images unusable by AI (see Zhao’s ‘GLAZE’ tool, and MIT’s 
‘photoguard’). Unfortunately, as Melissa Heikkila points out, many of these tools can be circum- 
navigated by simply taking a screenshot of the image for use in training (2023). Even Herndon and 
Dryhurst—generally positive proponents for the use of Al—are attempting to solve the problem of 
consent in image use through the creation of their page ‘spawning’, which provides artists with the 
opportunity to opt in or out of future training. The need for a solution to non consensual image 
training is necessary not only for visual artists, but also creators across many fields. At the time this 
article is being written, there are major protests taking place in the form of the Writers Guild and 
Screen Actors Guild of America strike that are centered around the appropriation and nonconsensual 
use of both actors’ and writers’ artistic property. While my own experiments do little to redress these 
growing concerns, I hope that my forays might expose some of the mechanics of working in this 
problematic field as an artist. 
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Loss and Lossiness 


As we pass through the uncanny valley—a recent tweet about Midjourney v5 by Bilawal Sidhu on 
March 17, 2023 proclaimed that we have ‘crossed the uncanny valley’ (Bilawal Sidhu 
[@bilawalsidhu])—the other side is a smooth, richly coloured and yet somehow featureless meadow. 
These high quality, or ‘rich’, images lack ‘poor’ images’ ‘link to the present’ (Steyerl 2009)—though 
the relationship between high and low quality images in machine learning is actually intertwined, 
with poor images forming an essential and effective component of model training (Wasielewski 
2023). What is lost, then, is a sense of /ossiness, but in the wrong areas: while the rich multimodal 
expression of dreams, glitchy content, and distinct working method of earlier AI work is lost, only a 
higher quality and more instantaneous image is gained. Lossiness is a term lifted from file compres- 
sion, and here indicates a perceptible aesthetic or trace of compression that has occurred (such as the 
characteristic warbling of a poorly compressed MP3). Ian Rothwell’s proposal that lossiness can be 
framed as a fundamentally queer encounter (2021) resonates with me: it is this queer weirdness in my 
early encounters with Al—and with dreams themselves—that is missing from this new workflow. 

As an artist, locating and deploying loss and lossines as productive creative processes has held great 
interest to me over many research-creation projects. For example, in earlier music works, I have 
created a multigenerational system for the oral transmission of textual and musical information that 
results in embodied ‘living scores’ (Nickel 2017), a process which foregrounds the productive and 
essentially creative nature of memory loss in iterated transmission (Nickel 2017). These works, as 
well as living scores by various other composers, ‘bloom’ in rich multimodal networks that flourish 
with the growth of community and repeated transmission (Nickel 2020). Somehow, loss seems like 
an essential component or byproduct of interfacing with our internal worlds and communicating 
them with others. 

Now, after reflecting on my Midjourney experiment nearly a month after making these images, 
my recall of my original dream has melded with its new AI counterpart, prompting me to wonder: 
what has been lost? And what was lost in the original translation of my dream to a written description 
from memory or the initial experience? Could any of these stages be transformed into generative, 
creative loss? 


Context and Comparisons (Scientific Practice vs. Creative Practice) 


[L.N] S. J., asa scientist and artist, do these ideas of imagery, narrative and replacement spark any 
resonances? 

[S. J. Abiodun]: There is a Jot to be said in reply to your thoughts about these ideas, especially in 
juxtaposing cognitive and creative perspectives. 


As ascientist, my approach and outlook on the increasing use of Al is centrally tied to its implica- 
tions within institutional research settings, specifically making use of machine learning algorithms, 
natural language models, iterative programming interfaces and ‘smart’ analysis pipelines. Researchers 
discuss these tools in the context of optimization and fine-tuning the precision of data interpretation. 

As an artist, more specifically one centered in visual storytelling and moving images, I have been 
intentional in studying and practicing organic, manually iterative synthesis, taking on a far more 
‘analog’ approach than my scientific practice may suggest. I do not necessarily have an aversion 
toward computational tools in art-making; in fact, recent projects such as ‘Computational Poetics’— 
a 2022 exhibition curated by Hannah B Higgins & David Familian at the University of California, 
Irvine— and ‘REKOGNIZE’— a 2017 triptych film by Bradford Young comprised of scenic pho- 
tography and matrices of metadata extracted from archival images— are just two examples of thrill- 
ing mergers of aesthetic and analytic practices, bringing life to the idea of ‘illustrating the iterative.’ 
However, I am driven by a meta-analytical desire to consider how processes of generative synthesis 


Art_ificial Intelligence: Dreams, Data, and Neuroaesthetics in the Age of AI | 147 


differ between scientific and artistic contexts. If I consider myself a creator, for instance, what 
underlies my concept of what it means to ‘create?’ If] am a scientific analyst, conversely, what 
motivates my desire to produce scholarly knowledge by interpreting existing or generated data? I 
juxtapose quantitative significance and qualitative nuance as a way of comparing practices seeking 
objective definition to those finding subjective meaning. 


Multimodal Data Frameworks and Dimensionality of the Input 


Thinking about the cognitive framework of it all, we can compare the quantitative vs. qualitative 
in the context of top down (goal-oriented) vs. bottom up (experience-driven) processing, and also in 
relation to the dimensions of raw experience/input vs. subsequent interpretation/output. Any phys- 
ics enthusiast will tell you that the process of transduction within a system—whether the transduction 
of energy, information, or experience—will result in some loss of its dimensionality, simply by virtue 
of the process of being mediated/transferred through some conduit or tool. The same applies in the 
context of our brains mediating interactions between perception and memory: when you go from 
an initial experience (crying while watching the first five minutes of Pixar’s Up), to memory consoli- 
dation (storing that memory for a rainy day), memory retrieval (recalling that scene when you see a 
balloon a week later), and reflection (recounting the sad scene to a concerned friend who notices you 
crying), the characteristic differences between initial and eventual narrative is influenced by subjec- 
tive, temporal, and modal contexts (Barber & Mather 2013, Schacter 2022, Zadbood et al 2022). 
Even when this process is occurring within a ‘contained system’, i.e our own minds, there is still a 
potential change in the fidelity of that information any time it is mediated or transduced, similar to 
the lossiness you described. This is the conundrum that arises when trying to approach bottom-up 
processing within a top-down framework. 

When asking this question in the context of dreams, I feel inclined to plainly declare that ‘dreams 
aren’t data’. This is a broad statement, in many ways, and would be directly contradicted by a 
moment’s consideration of the historical roots of dream interpretation evidenced in spiritual rituals 
and shamanic practices, research on consciousness and mind-wandering, and the foundations of 
psychoanalytic theory (Mfusi 1984, Barrett 2001, Domhoff & Scheider 2015, Christoff et al 2016). 
What I really mean is the following: multidimensional matter, like dreams, resists convenient 
datafication, and exists beyond reduction into representative values within a model-defined vector 
space. When trying to fit experiential or visceral knowledge (such as dreams, imagination, emotion, 
etc) within a probabilistic inference model, we should be critical of the importance placed on the 
descriptive value of our ‘data’ (Birhane 2021). Many expressive processes are egocentric (subjective, 
‘local’) by nature, and therefore lend themselves well to efforts to ‘interpret meaning’. When medi- 
ated through an allocentric (relative, ‘global’) framework, such as a pre-trained Al intermediate 
being used to ‘seek definition’ within a particular canon or knowledge schema, these data must be 
regarded with nuance. 

Furthermore, the aforementioned historical practices of dream interpretation all have a common 
thread: human mediation. We can understand dream interpretations as “output” generated by 
another human participant, rather than a mechanistically-trained model. When considering the 
synergetic relationship between medium and mediator (whether in the context of art, psychology, 
or society at large), one cannot assume that an artificial intelligence agent — regardless of how 
robustly trained or tested — can fully replace the contextual perspective (and access to expansive 
canon, relational experience, and associative knowledge) provided by a human interpreter. 


Interpretive processes and implications of the output 


[L. N.]: Given the way you frame dreams as resisting data-fication, I wonder why dreaming has 
been a central metaphor surrounding AI? 
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From Phillip K. Dick’s novel to Google’s 2015 DeepDream technology, which was quickly 
compared to dreaming and psychedelia (Hern 2015), Al and dreaming seem tightly intertwined. 
Using AI to analyze written textual accounts of dreams is a growing field (Fogli et Al 2020, Yu 2022, 
MacNamara et Al 2023). Google’s DeepDream paved the way for the use of dreaming as a meta- 
phor or image-generation—now used in MoMa’s exhibition to describe Refik Anadol’s Unsupervised 
(Davis 2023). The reverse is also true: processes involved in AI such as overfitting and noise are used 
by scholars such as Erik Hoel (2020) to describe human dreaming. Perceptions divorced from 
reality are often considered dream-like, and we’ve heard descriptions of AI ‘hallucinating’ when it 
creates counter-factual knowledge (Tung 2022). What can this parallel tell us about how AI relates 
to the narratives of our inner world? 

[S. J.]: As we navigate ever-evolving disciplines — both within creative and intellectual commu- 
nities and institutions — we are asking as practitioners, researchers, users and individuals, how to make 
sense of the gradual integration of Al into many aspects of our lives, creative and personal. We ask how 
we are meant to engage with these changing technologies, as tools of interpretation and analysis, or 
collaborators (and rivals?) in knowledge synthesis and meaning formation. And most notably, we 
debate about the ethical implications of automation in practices of creation, expression, and labor. 
Does using AI as a medium of generation and analysis (of art, of data, of experience and reality) 
encourage convergent interpretation, or is it a reductive lens through which formative experience 
is limited to formulaic equation? This ties back to the point I made above, regarding the difference 
between top-down vs. bottom-up guided processes. Any top-down model approach lends itself to a 
system of hierarchical evaluation: the system receives the information and biases its interpretations 
based on what it considers ‘significant value’, ‘best output’, and ‘highest quality’. 

In many research contexts, data quality optimization is achieved by tightly controlling experi- 
mental parameters (such as participant demographics, experimental setting, stimulus complexity, 
and scales of measurement) to ensure reliability and reproducibility of analyses and findings. This 
optimization perspective is a core component of many scientific/epistemological frameworks, but 
must be applied with caution when used to frame ‘narratives’ around real-world experiences and 
expressions that exist outside of institutional (read as: experimentally controlled and contrived) con- 
texts (Hasson 2020). 


3. Expressive flexibility: thinking about feelings and feeling through thoughts 


when we (i) think of neuroaesthetics 

as field, as practice, as framework 

{through;by} which we measure 

signal to noise ratio 

{through;by} which we explore 

the cognitive 
processes of {memory;perception;social cognition;emotion} 
et cetera 

the creative 
processes of experience; expression; exchange; exhibition} 
et cetera 

we subsume the two within a juxtaposition 

between aesthetic and epistemological 

between scientific and phenomenological 

between mechanistic and visceral 


A neuroaesthetic frame on methodological processes 


LL. N.]: How does your research in neurocinematics relate to the idea of measuring expressive 
information in a mechanistic way? 
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[S. J.]: When thinking about the goal of many tools of interpretation and analysis—such as the tools 
within cognitive research spheres to analyze various types of data (neural, behavioral, ecological, 
physiological)—there has already been a longstanding discussion of how we can broaden our research 
approach to avoid reductive conclusions that don’t generalize beyond experimental contexts (Neisser 
1982, Jolly & Chang 2019, Nastase 2020, Finn 2022). Within the field of neuroaesthetics—which 
examines the interplay between cognitive processes and aesthetic (narrative-driven) experience- 
this broadening occurs when scientists use media, such as film, music, and spoken narratives, as 
‘naturalistic’ experimental stimuli (Chen 2017, Sachs 2018, Nastase 2021) These studies aim to 
construct rich representations of cognitive phenomenon that are more ‘ecologically valid’, and are 
therefore more resonant with the dynamics of real world context. 

The memories, experiences and emotions that comprise our internal worlds are not only deeply 
complex, but deeply idiosyncratic (Chang 2021, Finn 2020). Due to their multimodal nature and 
contextual richness, these data often resist simplification into some of the frameworks of logical or 
temporal linearity that reduce heterogeneous sources of input to simple, definitive output, often 
achieved by trading off individual variability for group-level similarities (Chen 2017, Jolly & Chang 
2018, Nastase 2020, Horikawa 2020). 

Within naturalistic neuroscience, we already accept that our goal is not to eliminate dimension- 
ality loss entirely. This would be impossible, much like the inevitable loss of energy in any system 
when that energy is transduced. Rather, we aim to capture the components of the signal — whether 
of BOLD data or electromagnetic pulse — that supersede a predetermined threshold originating 
from equipment interference or data anomalies. This proportion is often referred to as our ‘signal- 
to-noise ratio’, and having a high SNR is integral in characterizing what pattern fluctuations are 
significant enough to draw robust conclusions (Nastase 2020, Hasson 2020). 

In many ways, neuroaesthetics is a practice in multimodal mosaic making, weaving together 
aesthetic appreciation and neural response (Cinzia & Vittorio 2009, Chatterjee & Vartanian 2014). 
It merges with naturalistic neuroscience through the integration of multidimensional inputs—neu- 
ral, behavioral, and physiological feedback—to construct temporally-flexible, higher-order repre- 
sentations of stimulus response (Saarimaki 2021). 


Extraction vs. expression 


Naturalistic experiments are meticulously designed to extract as much signal as possible from the 
inputs, and fed into models and analytical tools specifically designed to interpret complex data 
(Schindler 2017, Lettieri 2021, Heusser 2021, Saarimaki 2021). But even with these considerations 
in place, we are still faced with a crucial paradox: Using quantitative (goal-oriented, reductive) 
frameworks to interpret qualitative (salience-driven, expressive) information, trying to merge the 
worlds of mechanistic thinking and aesthetic expression. Our current tools work optimally when we 
can define discrete categories, measure along continuous scales, describe phenomena concisely and 
consistently. In a world where artistic appreciation and creativity are fueled by interiority, subjec- 
tivity and descriptive nuance, I argue that even our most complex analytical approaches will, at 
times, strugele to capture the finer dimensions of creative, emotive, visceral expression (Birhane 
2021). When you can’t always think through your feelings, and can’t always feel through your thoughts, 
artificial minds may find their signals crossed. 

Thus we reach an impasse, switching our gaze between a canvas doused in color and texture and 
the empirical frame through which we try to siphon it. For cognitive researchers, this framing is 
second nature. Yet we are constantly faced with the question of how far we assume our frame 
extends, how much of the texture we’re able to preserve, and how privy we are to what our empirical 
tools can reveal about internal, aesthetic, emotional worlds. 
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4. Artists, Automation and Rupture 


when we (i) think of 
the {art;science} of {science;art} 
what do we consider our medium 
{through;by} which we explore 
the cognitive 
mediation of internal vs. external states 
using invented tools 
while seeking definition 
the creative 
mediation of imagination vs. translation 
using invented forms 
while seeking meaning 


where does the dreaming droid fit 

in the matrimony between 
medium and mediator 
cognitive and creative 
practice and process 
imagination and invention 
neuro and aesthetic? 


[S. J.]: As interdisciplinarians and individuals living in the digital age, the frequency of our inter- 
facing with AI processes and tools in everyday life has amplified. When we think of AI as data-driven 
technology, how do we contextualize it in relation to art and society? 


Alas automation 


[L. N.]: I believe artists can play a key role in how we interface with AI, both in thought and 
practice. While I sometimes balk at being called a ‘storyteller’, as an artist I often enliven, articulate, 
and even speculate on the future of technology. When enough artists engage in these acts, we have 
the capability to deeply influence mainstream thought. One thing I find particularly fascinating as 
an artist is the capability to hold multiple viewpoints and rapidly shift between them to try and 
articulate a many-sided position—a capability that feels essential as we wade through the murky 
debates of AI usage, sentience and ethics. 

Your question was about AI as technology: while this question is currently culturally relevant and 
deeply storied (“Many Minds”), it is also historically intertwined with the arts. One thing that has 
changed about the technology is access to large amounts of data: visual and written art has—mostly 
non-consensually—become data to feed predictive and analytical models. In tandem, data has been 
art-ified. Artists now use data as a raw material in countless ways—for example, mapping, visualiza- 
tion, generative works, and institutional critiques. AI is woven into the fabric of many of our 
interactions, from the intentional and artistic to the automated and banal. Yet somehow, despite its 
long history, at the moment it still feels like AI is disrupting the status quo. 

Like many turning points of automation—digital video editing, music sampling—machine learn- 
ing disrupts traditional ideas of intentional artistic practice. These earlier turns should not be dis- 
counted in terms of their profound change in how artists work. But when machine learning begins 
to be perceived as Al, not only is the user’s sense of agency altered, but it joins with advances in 
networking such as the internet to disrupt a user’s perception of a boundaried ‘self’, introducing a 
new but artificial other. Many users become passively engaged in a process that was once available 
only to the highly skilled after years of institutional training. For example, not all people editing videos 
on TikTok consider themselves artists, and video editing is now embedded into a daily activity— 
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from which data is extracted—but some filmmakers will now have grown up learning their skills 
through TikTok. The same will happen with AI: we are scant months or years from the first AI- 
native artists finding their ways into university art programs and major institutions such as the 
Hollywood film industry. 

When considering this possible disruption to institutions, it may be useful to consider how artmaking 
currently resides in them. In university and conservatory art institutions, artmaking is seen as knowl- 
edge-making, especially considering the recent turn toward embodied practice (Spatz 2015). Part 
of being an artist-researcher is not only creating new works of art but describing why and how the 
context in which you make the thing matters. What does it mean for artists to collaborate and create 
with body-less intelligences—artificial or otherwise? 


Anthropomorphism, sentience, and making work in the duality 


When trying to articulate this position of AI in relation to intelligence and creation, we exist ina 
constant state of linguistic slippage where—whether we mean it or not—we cannot help but refer to 
Al using anthropomorphic metaphors. Companies such as Microsoft and OpenAl posit Al as an 
ever-present and ever-helpful anthropomorphic collaborator. Even the most challenging critiques 
of large language models fall into these characterizations, such as Chomsky’s description of ChatGPT 
and Bing ‘refusing’ requests and ‘deceiving’ the user (Chomsky et al 2023). The most compelling 
account I have read focuses on analyzing AI using narrative tropes rather than anthropomorphic 
ones—analyzing the output of LLMs as anthropomorphic only as related to the human-generated 
linguistic narratives used as input, such as the archetypes of classic literature or question-and-answer 
internet threads (Nardo 2023). Like the descriptions of LLMs as non-sentient stochastic parrots 
(Bender et al 2021), narrative-formed Waluigis (Nardo 2023), DMT-like Shoggoths (algekalipso 
2023), shamans (Allado-McDowell 2023), or chatbots, I believe that we must find ways to to speak 
about AI that do not liken it to the human brain. 

I often consider the benefits of imagining Al as a kind of early stage sentience, though still not 
human—less anthropomorphic and more cosmic, following ideas by authors such as K Allado- 
McDowell. Researchers such as Bender (2023) and Guest (2023) rightly point out that treating Al as 
human will lead to a gradual desensitization in how we treat other humans—essentially dehumaniz- 
ing humans as we humanize AI. In particular, because of the regular portrayal of Al as a woman, this 
will lead to the further dehumanization and degradation of women. We must avoid this path. I 
wonder whether we can shift our portrayal of Al from human to something beyond human, specu- 
lating on new forms of intelligence. This twist in perspective may allow us to expand western notions 
of personhood, helping us to recognize sentience in configurations beyond the human. Thinkers 
such as Jacy Anthis are beginning to explore these ideas through the creation of frameworks for 
considering the rights of AI as if it demonstrated characteristics of early sentience (2021, 2023). 
Anthis makes a compelling case that as humans we have not tended to recognize sentience easily, and 
that we may miss (and mistreat) early stage sentiences. On an artistic level, thinking and creating 
through the duality of AI-as-technological-tool and also a kind of beyond-human sentience has 
yielded highly creatively engaging processes and results for me and other artists. 

In their piece Some Must Watch While Some Must Sleep (2023), artist and writer Tanya Marquardt 
fine-tunes a GPT-3 chatbot on transcriptions of Marquardt’s ongoing sleep talking persona ‘X’. 
Previous to making the piece, Marquardt had captured countless recordings of their sleep talking, 
gradually realizing that the sleep talking belonged to a self seemingly distinct from their waking one, 
called ‘X’. In Some Must Watch, Marquardt invites audiences to send text messages as input to the 
fine-tuned model and receive over-night text transmissions. By doing so, Marquardt allows audi- 
ences to participate in this uncanny realization, calling into question autonomy, automation, and the 
dualities of fact and fiction. Here, AI becomes a self-serving and self-soothing tool, similar to S.J’s 
initial observation of users illustrating their dreams. AI also becomes a feedback tool that goes 
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beyond the analytical and enters the realm of the spiritual, as with Allado McDowell’s experiments 
with Al as hallucinogen. Similarly, artist-scientist Michelle Huang fine-tuned GPT-3 on her child- 
hood diaries, allowing her to converse with her past self (2022). Marquardt and Huang are engaging 
in what Holly Herndon and Mat Dryhurst call ‘identity play’ (2022). In Herndon’s holly+, she and 
Dryhurst create an AI model of Herndon’s voice, which is then also licensed out toa DAO of artists 
who both create using her likeness and collaborate on its worldly presence. These artists are engag- 
ing in AI playfully, holding space for and speculating on multiple possible realities and reflections of 
both their own internal and external worlds. 

[S. J.]: There is also a point to be made about how AI may provide a vehicle for constructing 
representations of selfhood and community for Queer and Trans individuals. Another component 
of the surge in Midjourney’s popularity in 2022 was a number of Trans and gender non conforming 
individuals mentioning how being able to generate images of themselves that mirrored their inter- 
nal representation provided a level of gender euphoria (Keller 2022). This echoes the affirming 
dynamics of ‘identity play’ you mentioned previously, while also offering a perspective of genera- 
tive Al as a potential tool for Queer visibility and liberation of marginalized peoples. This is a fresh 
take on the concept of utopic/euphoric inventions—also touched upon in the theory underlying 
Utopic Imagining and Glitch Feminism—where artists and individuals intertwine their selves, their 
data, their likenesses and their histories within digital/technological frames to create multifaceted 
interrogations of automation, selfhood and future decolonial realities (Russell 2020, Islam 2022). 


5. Art (Making) In The Age of Technocapitalism 


How do we ensure (within an ‘empirical’, ‘institutional’, ‘technocapitalist’ framework) 
That we understand 
(i was going to say heed) 
The implications of our ‘scientific’ inventions on aesthetic practice 
In a world where other institutions and populations and communities 
hold true to a different canon, 
set of rules, 
manner of practice, 
ethic of commodification/access, 


code of conduct 
> tveryone ~ 


we are participants in this age, whether willing or not, 
non coerced or coerced, but also without the rules of 
play, without informed consent 


© Everyone can reply 
eeroG die 


A performative exploration of Al representation 


[S. J.]: Now we come to the point where we ought to discuss the importance of context and 
positionality. The capacity of any research tool to ‘know’ and ‘define’ is directly contingent upon the 
knowledge and definition spaces instilled by its creators. We cannot think about Al as a knowledge 
apparatus separate from the context of who creates these technologies, where the data come from, how 
we determine what modalities and inputs these tools are trained on and deem ‘acceptable’, and what 
assumptions are made in these processes (Buolamwini & Gebru 2018, Raji et al., 2020). 

Zooming out from the current stream of conversation to a more historical perspective, we must 
also pause on our generally hypothetical approach to this dialogue to consider the very real histories 
of exclusion, prejudice, and oppression that frame how many tools for technological advancement 
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have been utilized. In a piece I wrote a few years ago, I detailed the implications of scientific racism 
and exploitation as it related to the field of neuroscience and Black communities (Abiodun 2019). 
To briefly revisit some of my main points, there are few ways to holistically evaluate the ‘progres- 
sive’ and ‘innovative’ potential of any discipline without considering that the very foundations of 
our institutions are rooted in epistemological violence, exploitative practice, and delegitimization 
of ‘alternative’ schools of culture, community and practice. This call to awareness resounds even 
louder when evaluating a space in which we are training automated tools—on large corpora of data, 
yes, but not without bias nor contrivance-to formulate conclusions about past, present, or future 
elements of our society at large. Rather than elaborate too much on the existing robust discussion of 
the Al and the ethics of positionality, I will offer an anecdotal example from an ‘exercise’ I engaged 
in for the sake of this discussion: 

When I decided to interface with Open Al’s image generating tool (DALL-E 2), I fed ita number 
of prompts to explore the specificity/flexibility of its output. I began with a prompt of ‘the future’, 
which yielded a series of images of suburban landscapes, images of crowds, and (textually inaccurate) 
poster-like images. 


Then I proceeded by prompting it with ‘the past’-yielding many black and white (many war-time- 
like) facsimiles, also many with crowds and backdrops alluding to particular historical time periods. 


oe 
LEiRys +3 


ke ISeve 2 
bape 


Then I pivoted, desiring to pry at the inner workings of the black box, and fed the tool ‘Artificial 
Intelligence’, to which it responded with the more stereotypical depictions of androgynous blue 
robotic figures surrounded by bites and bites of floating data and textual chunks, a known ‘blind spot’ 
in the field of AI ethics (Romele 2022). 


154 | JOURNAL OF COMPARATIVE LITERATURE AND AESTHETICS 


F 
f 
\ 
Ee 
E 


E 
t 


8 
i 
And then I pushed further, and fed the tool the prompt ‘self portrait’, curious as to whether it would 


generate similar techno-aesthetic images, or something else entirely. The AI tool returned with 
images of white men, stacked above the robotic portraits from the previous prompt. 


I ran this same prompt on DALL-E 2 and DALL E (concurrent versions of the same Open Al 
program), with ten iterations for each. 

For DALL-E 2, 80 total images were generated by its tenth iteration: several of racially-ambigu- 
ous, white-passing figures, several of gender-ambiguous individuals perceivable as ‘women’. Aside 
from these outliers, 90% of the images passed as ‘white men’: 
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For DALL-E, 40 total images were generated by its tenth iteration: (to my surprise) the vast major- 
ity of generated images were not of white men, but white women. After 10 iterations, roughly 28% 
of the generated images passed as ‘white men’ around 5% of images depicted racially ambiguous, 
dark-skinned figures (potentially perceivable as ‘non-white’). Besides these two categories, the other 
67% of the images were of figures that passed as ‘white women’. 


sedinears 


This anecdotal example is offered with the following subsequent consideration: What are the impli- 
cations of a technological future when the AI gazes through a lens of whiteness? It is a future in which we 
must critically consider how much our systems are dominated by narratives of white supremacy and 
cis-hetero-patriarchal hegemony. It is not a future we can deflect, it is a future that we are already 
within, and one that is accelerating at a rapid pace (Toole 2019, Toole 2021). Philosopher Dr. 
Briana Toole alludes to this phenomenon in her analysis on standpoint epistemology—that is how 
social identity and positionality directly affect what is considered/valued as ‘knowledge’ within a 
system—and its relation to epistemic oppression. Toole states the following: 


“marginalized knowers are largely excluded from the meaning-generating practices in which we develop 
és é : : Inns IK 

new conceptual resources. The result is that our intercommunal conceptual resources are often not suited to 

make sense of the experiences of the socially marginalized.” 


(From Standpoint Epistemology to Epistemic Oppression, Toole B, p. 609) 


Ifthe very knowledge instruments being used to illustrate our futures tend toward exclusivity in their 
portraiture, we are left wondering how exclusive these same tools will be in other iterations of 
meaning making, culture shaping, and knowledge building. Even with the ongoing discussion 
around the biases in Al training and output (Ntoutsi et al 2019, Ferrer et al 2021). We must ask how 
we can continually and intentionally root ourselves in narratives and epistemologies that directly 
critique white validation, and challenge biases in perspective and interpretation by still honoring the 
importance of dynamic representations of knowledge, reality, and meaning, 


Toward the speculative 


[L. N.]: Absolutely. Just like describing the sensations and perspectives lost when transforming 
dreams into data, what you’re describing is a kind of lossiness here too, one where perspectives 
beyond the white middle class are lost through the exclusion of non-white and marginalized scien- 
tists and data. Lossiness pervades so much of our modern interfacing with technology, both in 
positive and productive ways in the senses of glitch and transformation and also in more nefarious 
ways in modes of exclusion and bias. What routes can we explore that might move us out of those 
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exclusionary grooves into thinking generatively about technology and lossiness? One project I’m 
reminded of is the Indigenous Protocol and Artificial Intelligence Position Paper (Abdilla et al 
2020). This publication and conference imagines Al as a participant and tool embedded into various 
Indigenous communities, emphasizing the speculative and worldbuilding as tools for imagining 
more just futures of technology. I also took part in a year-long project led by Dylan Robinson called 
Decolonial Imaginings, which saw five settler-descended artists study Dylan’s book Hungry Listen- 
ing (2020) as well as create speculative scores surrounding decolonial sound practices. Dreaming 
about the future, thinking speculatively, and imagining artistic processes continue to form impor- 
tant tools to navigating AI beyond a predominantly white technocapitalist framework. 


6. Speculative Futures 


If we are trying to translate these things between these Spaces 
(rather than, in redundancy, resound similar gongs in our respective echo chambers) 
And understand how these tools can be used 


Then we must come to the table 

And consider our facets 

Where and how these terms are being used 

AI, ART, DATA, CRAFT, SCIENCE, MODEL 


And think of what picture 
What mosaic 
We dream of weaving for ourself(s), and our world(s). 


(poetic text by S.J.A) 


[L. N.] Speculative Event Score for Future Musics: Song For AI 


spin up an instance from the latent Space of your dreams 

[dodge the warped areas bulging out as badly obvious advertisements, slide around the awkwardly aligned 
water table formed from corporate value systems, jump over the deep grooves of the narratives of western 
human literature and 2000s era reddit posts] 

listen to your waking life through the prismatic kaleidoscope ofa new tarot 


feel the ghostly impulses in your body, the combination of generations of embodiment and years of training 
[calm the twitching nerves of new glitches, quell the suggested movements without clear motivations, reroute 
requests for interruption] 

stretch your vocal chords, uncoiling the wet springs of your body 


sing a duet with your dream data 

[sow the soil with recordings of your singing, respect the bones of data stolen from those without input, reject 
the gleaming and bloated filtered models tailored for the consumerist version of you] 

hear the soft predictions and suggestions of your past sleeping sonic pathways crackle and leapfrog ahead of 
you, and navigate them, singing to decide what the sounds of the future will be 


[S. J.] Speculative Triptych film script: A&&&l (u, me, and steve) 
SPEC SCRIPT: A&E&EI (u, me, and steve) 


AG&&E is an experimental triptych film cocreated by artist sa:de and artificial intelligence agent STEV-E, a 
hypothetical tool capable of cinematic generation (sound, image, narrative). In this film, each member is tasked 
with crafting their translation of two written poems (VIS-01, VIS-02), and a sonic poem (AUD-01), which 
will be presented simultaneously, in triptych composition, while overlayed with the lyrics and poems from the 
source materials. 

This film explores the interplay of imagination and invention, creation and computation, and presents the 
‘members’ as collaborators, rather than as forces in opposition, in the process of composition and translation. 
This triptych draws on several core references: 
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* Symbiopsychotaxiplasm: Take One by Dir. William Greaves (1968) 
* Flowers for Algernon (1966) by Daniel Keyes 

* REROGNIZE by Dir. Bradford Young (2017) 

* Listening to Images (2017) by Tina Campt 

* Godspeed by Dir. sa:de (2020) 

* AMPERSEND: act i by Dir sa:de (2022) 


VIS-01: One Art (Elizabeth Bishop) 


The art of losing isn’t hard to master; 

so many things seem filled with the intent 

to be lost that their loss is no disaster. 

Lose something every day. Accept the fluster 
of lost door keys, the hour badly spent. 

The art of losing isn’t hard to master. 


Then practice losing farther, losing faster: 
places, and names, and where it was you meant 
to travel. None of these will bring disaster. 


I lost my mother’s watch. And look! my last, or 
next-to-last, of three loved houses went. 
The art of losing isn’t hard to master. 


I lost two cities, lovely ones. And, vaster, 
some realms I owned, two rivers, a continent. 
I miss them, but it wasn’t a disaster. 


—Even losing you (the joking voice, a gesture 
I love) I shan’t have lied. It’s evident 

the art of losing’s not too hard to master 

though it may look like (Write it!) like disaster. 


VIS-02: On occasion, we produce history, the present’s surprise (Erica Hunt) 


We measure speed by the absence of interruption. 

We measure safety by the string of near misses. 

We anticipate the end by who is telling the story. 

At this time of night, there is a machine that calls you by name and talks to other machines where you live, 
where you dance by your fingertips over the globe, an address at a time, day into night. 


This machine feigns a reckless intimacy with you, corrects your spelling errors, as if reading your mind, but 
skips over others, like replacing eros with errors and spiraling with spelling. 


The other machines are being dismantled. They tell a different story. They draw the attention of the curious, 
the ones willing to go out of their way—past the land of leaves drop, the valley of forgetting, over a bridge too far, 
past the flag of fictitious victory over to the corner of vanquished subjects, where common love is almost concealed. 


AUD-01: Dream (Ryuichi Sakamoto). 


listen to audio at https://www.youtube.com/watch?v=uW Kh_7Tv YaqE 
rest in peace, ryuichi sakamoto. 


7. Concluding Thoughts 


Dreams are woven to the very fabric of our existence, whether those threads are tied to worlds of 
imagination (creative, expressive, symbolic) or invention (generative, analytic, definitive). We 
view our preceding dialogue as an exposition of the grooves of our collective consciousness (and 
subconsciousness), an intersection of creative exercise and intellectual exploration that allows us to 
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remain engaged through a period of complex technological and societal change. What rises up 
through all our experiments and expositions is a question: how do we account for the lossiness? What 
is lost when we move from the chorus of saxophones wailing into the night, to the symphony of 
sounds bursting from a record player as it dances along the grooves of a pressed vinyl, to the peaks 
and troughs dancing across a monitor as the sound is digitized, to the iterative loops of a machine 
learning model as it is trained on tune after tune? What is sacrificed when we transcribe dreams from 
memory to word, from text to image? What is the cost of assuming our technologies are universally 
representative, comprehensively inclusive, and reliable translators of the salient components of a 
sentient existence? And how is this lossiness a fundamental texture of our present encounter with the 
mediated world? Art_ificial intelligence becomes art_ifact, where lossiness is ingrained into every 
object, whether material or technological. 

Thus we try to speak to how our concept of artificial intelligence factors into our present— how 
our current creative, intellectual and social economies have evolved to incorporate such tools into 
the minutiae of operation and execution. In engaging with each other, we aim to explore the 
philosophical, empirical and ethical frameworks within which we, as members of the general popu- 
lation and diverse intelligences of this planet, engage with artificial intelligences in the realm of 
research, artistic process, and everyday life. We are artists, researchers, and interdisciplinary scholars 
whose personal curiosities and experiences with the topic matter(s) at hand—namely creative prac- 
tice, empirical process, and their intersections with Al-have fueled a desire to create a capsule 
representative of recent thoughts, conversations, and points of interest in the general discourse(s) 
around the relevant themes. We are met with the conundrum that there is no capsule of time in 
which we can say that this interaction started, nor can we say where it will end or possibly go. And 
yet, we continue to dream, we continue to discuss, we continue to train and test and retest our own 
bases of knowledge practices, as we come to understand how (and where, and when) our techno- 
aesthetic future may unfold. 
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